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ABSTRACT 


This  thesis  documents  the  collection  and  analysis  of  experimental  data 
used  to  compare  and  contrast  three  methods  of  evaluating  attribute 
choices  that  have  no  natural  measurement  basis.  Attribute  choice 
evaluation  is  prerequisite  to  ranking  and  evaluating  multi-attribute  decision 
alternative  sets.  Data  for  the  methods  center  on  subjective  inputs  from 
subject  matter  experts.  The  particular  experiment  collected  response  data 
from  27  experts  in  the  United  States  Navy  LAMPS  (Light  Airborne  Multipurpose 
System)  helicopter  community  attached  to  the  Naval  Postgraduate  School. 
The  pilots  compared  three  helicopter  systems  (attribute  choices)  in  each  of 
four  system  categories  (attributes):  vy/eapons,  navigation  systems, 
communication  systems,  and  sensors.  The  complete  procedure  would 
evaluate  every  feasible  helicopter  system  suite  (decision  alternative  set), 
each  set  composed  of  one  attribute  choice  from  every  attribute,  facilitating 
ordinal  ranking  of  the  sets.  Thesis  results  present  consistency  analyses  of  the 
experts'  responses  within,  and  between,  the  three  methods  of  determining 
attribute  choice  values. 
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EXECUTIVE  SUMMARY 


Military  decision  makers  are  constantly  challenged  to  make  optimal 
choices  between  competing  alternatives.  This  challenge  is  made  even 
more  demanding  in  that,  often,  competing  alternatives  are  very  complex 
and  have  no  natural  measurement  basis  from  which  to  conduct  direct 
comparisons.  By  sufficiently  dividing  each  of  the  complex  alternatives  into 
characteristics,  called  attributes,  common  to  all,  or  at  least  most,  of  the 
alternatives,  direct  comparisons  and  evaluations  of  each  alternative's 
particular  attribute,  called  attribute  choices,  might  be  made.  By 
weighting  the  importance  of  each  of  the  attributes,  a  value  for  each  of 
the  competing  alternatives  can  be  calculated  and  direct  quantitative 
comparisons  made. 

This  thesis  focuses  on  the  first  major  step  of  the  overall  process: 
evaluating  attribute  choices  with  no  natural  measurement  basis.  Three 
methods  of  determining  attribute  choice  values  which  use  subject  matter 
expert  subjective  inputs  were  compared  and  contrasted.  Experiment 
response  data  from  27  experts  in  the  United  States  Navy  LAMPS  (Light 
Airborne  Multipurpose  System)  helicopter  community  were  examined  for 
response  consistency  within  and  between  the  three  methods. 
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The  three  methods  examined  to  evaluate  attribute  choices 


involved  direct  evaluation  via  a  linear  scale,  and  indirect  evaluation  via 
pairwise  comparisons  and  indifference  probabilities.  Derived  data  values 
were  plotted  versus  directly  obtained  values.  The  amount  and 
characteristic  behavior  (bias)  of  the  derived  values'  dispersion  from  the 
directly  obtained  values  within  and  between  each  of  the  methods  were 
calculated  and  compared. 

The  smaller  the  magnitudes  of  dispersion  and  bias,  the  more 
consistent  the  responses.  The  intra-method  consistency  checks  in  the 
table  indicate  that  while  the  pairwise  comparison  method  showed  the 
least  dispersion,  it  was  more  susceptible  to  bias  than  the  indifference 
probability  method.  The  same  conclusions  seemingly  can  be  drawn  from 


Intra-method  Consistency  Checks 

Dispersion,  d........ 

Bias, 

Pairwise  Comparisons 

0.0299 

+0.1401 

Indifference  Probabilities 

0.1851 

+0.0420 

Inter-method  Consistency  Checks 

Dispersion,  d„^ 

Biasr  b„^ 

Marking  a  Linear  Scale  vs 

Pairwise  Comparisons 

0.0361 

-0.2264 

Marking  a  Linear  Scale  vs 

Indifference  Probabilities 

0.2968 

-0.0862 

Pairwise  Comparisons  vs 

Indifference  Probabilities 

0.3962 

-0.2052 
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the  inter-method  consistency  checks;  however,  method  interaction 
prevents  definite  inferences  from  being  made. 

Additional  consistency  checks  seemed  to  imply  the  experts  were 
risk  prone  when  they  perceived  there  was  little  to  loose  and  much  to 
gain.  Conversely,  the  experts  were  risk  averse  when  they  perceived  there 
was  much  to  loose  and  little  to  gain. 
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i.  INTRODUCTION 


A.  THESIS  SCOPE  AND  GOALS 

This  thesis  documents  the  collection  and  analysis  of  experimental 
data  used  to  compare  and  contrast  three  methods  of  evaluating 
attribute  choices  that  have  no  natural  measurement  basis.  Attribute 
choice  evaluation  is  prerequisite  to  ranking  and  evaluating  multi-attribute 
decision  alternative  sets.  Data  for  the  methods  center  on  subjective 
inputs  from  subject  matter  experts.  The  particular  experiment  collected 
response  data  from  27  experts  in  the  United  States  Navy  LAMPS  (Light 
Airborne  Multipurpose  System)  helicopter  community  attached  to  the 
Naval  Postgraduate  School. 

These  LAMPS  pilots  were  asked  to  compare  three  helicopter  systems 
(attribute  choices)  in  each  of  four  system  categories  (attributes);  weapons, 
navigation  systems,  communication  systems,  and  sensors.  The  complete 
procedure  would  evaluate  every  feasible  helicopter  system  suite 
(decision  alternative  sets),  facilitating  their  ordinal  ranking.  It  should  be 
emphasized  that  this  thesis  concentrates  on  consistency  analyses  within 
and  between  the  three  methods  of  determining  attribute  choice  values. 

It  is  not  the  goal  nor  intent  of  this  thesis  to  evaluate  LAMPS  helicopter 
system  suites. 

B.  THESIS  OUTLINE 

Chapter  II  provides  definitions  of  multi-attribute  decision  alternative 
sets,  attributes,  and  attribute  choices,  and  an  overview  of  how  these 
elements  are  combined  to  evaluate  decision  alternative  sets.  The 
employment  of  subjective  inputs  from  subject  matter  experts  is  then 
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discussed.  The  bulk  of  the  chapter  concentrates  on  describing  the  three 
attribute  choice  evaluation  methods.  Techniques  used  to  apply  the 
methods  are  discussed,  as  are  ways  to  conduct  intra  and  inter-method 
consistency  checks. 

Chapter  III  highlights  the  experimental  approach  used  to  collect 
the  thesis  data.  Also  included  is  a  brief  discussion  of  the  history  and 
current  mission  applications  of  LAMPS.  This  discussion  is  meant  to  give  the 
reader  unfamiliar  with  LAMPS  some  insight  into  the  experimental  survey 
setup  and  question  design.  In  view  of  the  broad  range  and  variety  of 
missions  performed  by  LAMPS,  the  importance  of  and  reasoning  behind 
presenting  the  pilots  with  a  standardized,  albeit  generalized,  mission  are 
also  discussed.  The  specific  attributes  and  attribute  choices  used  for  the 
experiment  are  then  presented,  along  with  samples  of  the  81  possible 
helicopter  system  suites  that  could  be  created.  The  chapter  finishes  with 
an  overview  of  the  survey  response  form  and  guide,  and  a  discussion  on 
the  actual  administration  of  the  surveys. 

The  methods  used  to  analyze  the  data,  specifically  intra  and 
inter-method  consistency  checks,  are  described  in  Chapter  IV. 
Consistency  check  plots,  along  with  the  two  parameters  used  to 
quantitatively  measure  consistency,  are  derived  and  results  for  five  main 
data  subsets  are  presented.  Tables  summarizing  the  consistency  check 
parameters  for  28  various  data  subsets  conclude  the  chapter. 

Chapter  V  concentrates  on  interpreting  the  consistency  check 
parameters  developed  in  Chapter  IV  for  the  five  main  data  subsets. 

Some  consistency  checks  that  yielded  unexpected  results  are  analyzed  in 
greater  detail  and  utility  function  theory  applied  in  an  attempt  to  explain 
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the  results.  The  thesis  closes  with  discussions  on  method  applicability, 
experiment  conclusions  and  suggested  areas  for  future  study. 
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II.  MULTI-AHRIBUTE  DECISION  ALTERNATIVE  SETS  WITH  NO  NATURAL 

MEASUREMENT  UNITS 


Chapter  II  opens  by  providing  definitions  of  multi-attribute  decision 
alternative  sets,  attributes,  and  attribute  choices,  and  an  overview  of  how 
these  elements  are  combined  to  evaluate  decision  alternative  sets.  The 
employment  of  subjective  inputs  from  subject  matter  experts  is  then 
discussed.  The  bulk  of  the  chapter  concentrates  on  describing  the  three 
attribute  choice  evaluation  methods.  Techniques  used  to  apply  the 
methods  are  discussed,  as  are  ways  to  conduct  intra  and  inter-method 
consistency  checks. 

A.  DEFINITIONS 

To  fully  comprehend  the  evaluation  methods  described  later  in  this 
chapter,  the  following  definitions  should  be  thoroughly  understood: 

•Multi-Attribute  Decision  Alternative  Set:  Also  called  a 
decision  alternative,  alternative  set,  or  set,  a  multi-attribute 
decision  alternative  set  is  a  complex  object  upon  which  a 
decision  maker  wants  to  place  a  quantitative  value  in  order 
to  directly  compare  it  with  similar  complex  objects.  For 
example,  an  auto  dealership  may  present  a  perspective  car 
buyer  (i.e.,  a  decision  maker)  with  dozens  of  alternative  sets 
(i.e.,  cars)  that  he  wants  to  evaluate. 

•Attribute:  An  attribute  is  an  alternative  set  characteristic 
common  to  all  (or  at  least  most)  of  the  sets  considered  for 
evaluation  and  ranking.  For  example,  attribute  candidates 
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for  new  cars  could  include  gas  mileage  and  exterior  color. 
Note  that  some  attributes  (e.g.,  gas  mileage)  are  much 
easier  to  quantify  than  others  (exterior  color). 

•Attribute  Choice:  An  attribute  choice,  or  choice,  is  the 
particular  item  or  characteristic  of  a  certain  attribute  in  each 
alternative  set.  For  the  car  example,  attribute  choices  for 
the  exterior  color  attribute  may  be  green,  red,  and  black  for 
some  car  models,  and  silver,  dark  blue,  red,  and  turquoise  for 
others.  The  gas  mileage  attribute  choice  for  one  model  of 
car  might  be  1 7.9  miles  per  gallon  (mpg)  and  24.1  mpg  for  a 
different  model.  Note  that  the  number  of  attribute  choices 
need  not  be  the  same  for  each  attribute  (six  exterior  color 
attribute  choices,  but  only  two  gas  mileage  choices).  Also 
note  that  not  all  alternative  sets  may  be  available  or  feasible 
(the  dealer  may  not  have  a  red  car  that  gets  24.1  mpg). 

B.  PROBLEM  STATEMENT  AND  FORMULATION 

This  thesis  examines  and  checks  the  consistency  of  three  attribute 
choice  evaluation  methods  that  are  prerequisite  for  evaluating 
alternative  sets  that  have  no  natural  basis  for  comparison.  The  absence 
of  a  natural  comparison  basis  creates  questions  of  how  to  place  an 
overall  value  on  alternative  sets  and,  ultimately,  how  to  consistently 
choose  the  best  alternative  set. 

Rather  than  try  to  compare  alternative  sets  considering  all  attributes 
simultaneously,  the  sets  are  broken  down  into  single  attribute  choices 
upon  which  the  methods  are  then  used  to  derive  attribute  choice  values 
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and  attribute  weighting  factors.  From  these,  an  overall  value  for  each 
alternative  set  is  then  calculated. 

Assume  that  there  are  m  alternative  sets  and  that  each  alternative 
set  has  n  attributes.  The  alternative  sets  are  indexed  by  j  =  1 , 2, ...,  m,  and 
the  attributes  by  i  =  1 , 2, ...,  n.  Let  V,(j)  be  the  amount  of  value  obtained 
from  attribute  i  in  alternative  set].  The  overall  value  of  set]  is  assumed  to 
be  given  by  Equation  (1 ): 

V(])  =  a|  Vj  (])  for  any  ]  =  1,2, ...,  m,  where  (1 ) 

V(])  =  overall  value  for  alternative  set  ]; 

ttj  =  weighting  factor  for  attribute  i;  a|  Is  the  amount  of  set  ]’s 
overall  value  obtained  from  a  unit  of  attribute  i; 
a,  >  0  if,  for  attribute  i,  bigger  is  better  [Ref.  1 ,  Table  7.2];  and, 
tti  <  0  if,  for  attribute  i,  smaller  is  better;  and, 

V|(])  =  value  of  attribute  i  for  alternative  set  ]. 

Given  the  V,(])'s  and  oCj 's,  the  overall  numerical  values  for  each  of 
the  alternative  sets,  the  V(])'s,  can  be  calculated.  Once  the  V(])'s  have 
been  found,  they  can  be  ranked  to  find  the  alternative  set  with  the 
greatest  overall  value.  Mathematically  the  ob]ective  Is  to  find 

Max .  {  VO)  =  a,  V,  0)  +  V^O)  +  +  a,  V„0) }’  •  (2) 


'  Appendix  A  provides  a  more  in-depth  discussion  of  the  nomenclature  and 
indexing  relationships  used  throughout  this  thesis.  If  the  reader  continues  to  have 
questions  regarding  the  interrelationships  of  Equation  |1)  and  (2)  components,  he 
should  refer  to  Appendix  A  at  this  time. 
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Numerical  values  for  each  of  the  V,(])'s  and  Oi 's,  however,  are 
generally  not  readily  available.  Before  the  alternative  sets  can  be 
ranked,  these  values  must  be  determined. 

This  thesis  takes  an  experimental  approach  to  test  decision  maker 
response  consistency  within  and  between  three  methods  for  deriving  the 
V,(j)'s.  For  future  study,  data  was  also  collected  to  examine  expert 
response  consistency  in  deriving  's. 

C.  SUBJECT  MAHER  EXPERT  SUBJECTIVE  INPUTS 

Deriving  a  value  for  an  attribute  choice  may  be  a  matter  of  taking 
a  simple  physical  measurement,  such  as  temperature  or  volume.  Many 
attributes,  however,  do  not  lend  themselves  as  readily  to  evaluation,  or 
are  very  time  consuming  or  expensive  to  evaluate.  Attribute  weighting 
factors  are  even  less  easily  obtained. 

This  thesis  compares  several  methods  for  deriving  attribute  choice 
values.  The  methods  make  use  of  subjective  inputs,from  subject  matter 
experts  (also  called  decision  makers,  or  experts).  For  each  attribute, 
experts  are  asked  to  rank  the  attribute  choices  from  the  one  with  the  most 
preferred  level  to  the  one  with  the  least  preferred  level. 

Having  ordinally  ranked  the  attribute  choices  for  each  attribute, 
experts  are  asked  to  compare  the  choices  in  the  following  three  ways: 

For  each  attribute: 

•  assign  a  value  to  each  attribute  choice  from  a  linear  scale 
(Method  1); 

•  make  pairwise  comparisons  of  attribute  choices  by 
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assigning  a  number  to  the  ratio  of  their  values  (Method  2); 
and, 

•  find  an  indifference  probability  for  a  gamble  between  the 
best  and  worst  attribute  choices  and  a  certainty  of 
obtaining  an  intermediate  attribute  choice  (Method  3). 

Using  Method  1  the  V|(j)'s  are  obtained  directly.  Methods  2  and  3 
were  designed  to  build  in  consistency  checks  within  each  method. 

Experimental  results  are  presented  in  Chapter  IV  that  demonstrate 
the  levels  of  consistency  within  Methods  2  and  3,  followed  by 
inter-method  consistency  checks  between  Methods  1  and  2, 1  and  3.  and 
2  and  3. 

D.  METHOD  STRUCTURES  AND  CONSISTENCY  CHECKS 

For  each  attribute,  an  expert  is  asked  to  provide  a  series  of 
subjective  inputs  as  to  which  choice  possesses  the  most  preferred  level  of 
that  attribute,  the  second  most  preferred  level,  and  so  on  down  to  the 
attribute  choice  with  the  least  preferred  level  of  that  attribute.  The  expert 
then  uses  the  three  methods  to  quantify  the  degree  to  which  each 
choice's  attribute  level  differs  from  the  remaining  choices.  Finally,  these 
responses,  and  those  to  develop  attribute  weighting  factors,  could  be 
applied  to  evaluate  decision  alternatives.  Again,  this  thesis  does  not 
attempt  to  evaluate  decision  alternatives,  rather,  the  degree  of  expert 
response  consistency  within  and  between  the  three  attribute  choice 
evaluation  methods  is  examined. 
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1 .  Data  Acquisition  Steps;  Comparison  Methods  and  Consistency 
Checks 

A  series  of  responses' is  required  of  the  expert  in  order  to  collect 
data  to  derive  attribute  choice  values. 

a.  Step  1:  Attribute  Choice  Subjective  Rankings 

For  each  attribute,  the  expert  ranks  the  attribute  choices  from 
the  one  with  the  most  preferred  level  (i.e.,  the  best  choice)  of  an 
attribute,  to  the  one  with  the  least  preferred  level  (i.e.,  the  worst  choice) 
of  that  attribute.  For  example:  if  the  attribute  under  consideration  is  gift 
stature,  and  the  attribute  choices  are  a  $5.00  gift,  $20.00  gift,  or  $100.00 
gift,  then  the  expert  would  likely  rank  the  $100.00  gift  attribute  choice  as 
having  the  most  preferred  level  of  gift  stature  attribute,  the  $20.00  gift  as 
having  less  of  the  attribute  than  the  $100.00  gift,  but  more  than  the  $5.00 
gift,  and  the  $5.00  gift  last  for  having  the  least  preferred  level  of  the 
attribute  gift  stature.  Symbolically,  $1 00.00  gift  >-  $20.00  gift  $5.00  gift, 
where  the  symbol  >-  indicates  the  choice  to  the  left  of  the  symbol  is 
preferred  to  the  choice  to  the  right  of  the  symbol.  The  symbol  does  not 
represent  greater  than. 

Responses  for  Step  1  are  inherently  consistent.  They  would 
become  inconsistent  only  if  the  expert  later  decided  his  initial  rank  order 
did  not  correctly  represent  his  opinion.  If  the  expert  changes  his  initial 
rank  order  before  he  completes  all  the  data  collection  steps,  the  data 
collection  process  must  be  restarted  at  Step  1 . 

b.  Step  2:  Marking  a  Linear  Scale  (Method  1) 

The  expert  is  next  asked  to  quantify  the  attribute  choices 
using  the  three  methods  discussed  earlier  in  Section  C  of  this  chapter. 
Using  the  first  method,  designated  Method  1 ,  the  expert  derives  attribute 
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choice  values  directly  by  marking  a  linear  scale  labeled  from  zero, 
representing  an  attribute  choice  with  an  absolutely  worthless  value,  to 
100,  representing  an  attribute  choice  with  the  best  possible  value. 

Figure  1  shows  an  example  of  such  a  scale  and  a  possible  set  of  responses 
to  the  gift  stature  attribute  example  introduced  in  the  previous  section. 


Figure  1 .  Example  of  Method  1  (Step  2),  Marking  a  Linear  Scale. 


Method  1  responses,  considered  by  themselves,  are 
inherently  consistent  and  would  become  inconsistent  only  if  the  expert 
changes  his  rank  ordering  established  in  Step  1 . 

c.  Step  3:  Attribute  Choice  Pairwise  Comparisons  (Method  2) 
The  second  method  by  which  the  expert  is  asked  to  quantify 
the  attribute  choices,  designated  Method  2,  is  a  process  of  painA^ise 
comparisons  between  all  attribute  choices  within  each  attribute.  This 
method  provides  attribute  choice  values  indirectiy  via  ratios  of  attribute 
choice  values.  The  expert  is  asked  to  give  his  opinion  on  the  relative 
value  of  how  much  better  (more  valuable,  more  preferred,  etc.)  each 
best  attribute  choice  is  over  the  attribute  choice  with  the  second  best 
level  of  the  attribute,  third  best,  and  so  on  down  to  the  attribute  choice 
with  the  worst  level  of  the  attribute,  as  ranked  during  Step  1 .  He  then 
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compares  the  second  best  to  the  third  best  and  so  on  down  to  the  worst. 
This  process  is  continued  until  all  possible  pairwise  comparison 
combinations  of  attribute  choices  of  each  attribute  are  considered.  The 
expression  for  the  pairwise  comparison  ratio  is  shown  in  Equation  (3): 

■'ki  ~  ^  ”  ratio  of  the  values  of  attribute  choices  k  and  I,  where  (3) 
V,^  =  value  of  attribute  choice  k,  and 
V,  =  value  of  attribute  choice  I,  and 
V  >  V 

''k  ''l  • 

If  K|  is  the  number  of  attribute  choices  within  attribute  i,  then  the  required 

K!  Ki(Kj-l) 

number  of  comparisons  is  2j(|^.'l2)!  =  —  • 

Figure  2  demonstrates  an  example  set  of  pairwise 
comparisons  (Step  3,  Method  2)  with  the  three  attribute  choices  for  the 
attribute  gift  stature  discussed  earlier. 

A  technique  for  checking  the  expert's  Method  2  response 
consistency  starts  by  plotting  data  on  an  (X,  Y)  plot.  Let  r,^,  and  r^,^  be 
the  expert's  responses  for  the  pairwise  comparison  ratios  of  the  best  versus 
intermediate,  intermediate  versus  worst,  and  best  versus  worst  attribute 
choices,  respectively.  Now  plot  the  expert's  response  for  r^,^  on  the 
X-coordlnate.  From  Equation  (3)  we  know  r^,)  =  and  r;^  =  The 
product  of  these  two  ratios  reduces  to  a  derived  value  for  r^^^  and  is 
plotted  on  the  Y-coordinate.  If  the  expert  is  perfectly  consistent  in  his 
responses,  ~  plotted  data  point  will 

lie  on  a  line  through  the  origin  of  slope  1 .0.  A  quick  visual  consistency 
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check  then  notes  the  displacement,  if  any,  of  the  plotted  data  point  from 
the  line  drawn  through  the  origin  with  slope  1 .0. 


Give  your  quantitative  opinions  on  how  much  better  the  best  attribute 
choice  is  over  the  intermediate  and  worst  attribute  choices,  and  how 
much  better  the  intermediate  attribute  choice  is  over  the  worst. 

•Attribute:  Gift  Stature 

•Attribute  Choices:  Best:  $100  gift  Intermediate:  $20  gift  Worst:  $5  gift 


Expert's  responses 


Ratio  of  best  to  intermediate:  r^  =  VJ  V, 


•The  (Best)  $100  gift  is  5  times  better  (more  valuable)  than  the  (intermediate)  $20  gift. 


•The  (Best)  $100  gift  is  19  times  better  (more  valuable)  than  the  (worst)  $5  gift. 
•The  (intermediate)  $20  gift  is  3.5  times  better  (more  valuable)  than  the  (worst)  $5  gift. 


Figure  2.  Example  of  Method  2  (Step  3),  Attribute  Choice  Pairwise 
Comparisons. 


Quantitative  measures  of  response  consistency  involve 
calculating  the  vertical  bias  and  mean  squared  difference  of  the  data 
points  from  the  slope  1 .0  line.  Equations  for  calculating  these  measures 
are  discussed  in  Chapter  IV,  Section  A.  Figure  3  shows  a  consistency 
check  plot  of  r^^^  r.^  versus  r^,^  for  the  example  data  in  Figure  2. 

d.  Step  4:  Attribute  Choice  indifference  Probabiiities 
(Method  3) 

Method  3  (Step  4)  is  the  third  and  final  method  used  to 
quantify  the  expert's  attribute  choice  preferences,  and  involves  acquiring 
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the  expert's  attribute  choice  indifference  probability.  Perhaps  the  easiest 
way  to  define  an  indifference  probability  is  with  a  decision  tree. 

Figure  4  shows  a  simple  decision  tree,  or  decision  sapling 
[Ref.  1  ].  The  square  node  on  the  left  side  of  the  diagram  is  a  decision 
node,  the  round  one  is  a  chance  node,  and  the  diamond  shaped  nodes 
are  result  nodes.  The  result  node  labels  V. ,  V;,  and  are  the  values  of 
the  best.  Intermediate,  and  worst  attribute  choices,  respectively,  for  a 
given  attribute.  The  decision  maker  starts  at  the  decision  node  and 
moves  from  left  to  right  along  one  and  only  one  path  until  he  reaches  a 
result  node.  There  is  no  risk  involved  for  the  expert  if  he  chooses  to  follow 
the  upper  branch.  Upon  reaching  the  result  node  he  will  receive  a  value 
Vj.  He  may,  on  the  other  hand,  choose  to  take  the  lower  branch.  Upon 
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Figure  4.  Sample  of  a  Decision  Sapling. 


reaching  the  chance  node,  there  is  a  chance,  p,  he  will  take  the  upper 
branch  out  of  the  chance  node  and  receive  a  value  or  a  chance 
(1  -  p)  that  he  will  follow  the  lower  branch  and  receives  a  value  V^.  The 
indifference  probability  is  the  value  of  p  for  which  the  decision  maker  is 
indifferent  between  taking  the  gamble  and  taking  the  riskless  alternative. 

Figure  5  shows  an  example  of  a  type  of  problem  that  could 
be  posed  to  an  expert  to  determine  his  indifference  probability.  The 
example  uses  the  same  attribute  (gift  stature)  and  attribute  choices 
($100.00  gift  (best);  $20.00  gift  (intermediate);  and  $5.00  gift  (worst))  as  in 
previous  examples. 

A  technique  for  checking  consistency  within  Method  3 
requires  two  responses  for  each  attribute.  The  expert  first  provides  a 
response  to  the  original  indifference  probability  problem  as  demonstrated 
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^ •You  currently  have  the  (intermediate)  $20.00  gift.  You  may  keep  it,  or  you 
may  use  it  to  gamble  for  the  (Best)  $100.00  gift.  If  you  gamble  and  win,  you 
get  the  (Best)  $100.00  gift.  If  you  gamble  and  loose,  you  get  the  (worst)  $5.00 
gift.  If  you  do  not  gamble  at  all,  you  keep  the  (intermediate)  $20.00  gift. 


•What  is  the  minimum  percentage  chance  of  winning  the  (Best)  $100.00  gift 
you  are  willing  to  accept  in  order  to  take  the  gamble? 


Expert's  Response,  (p  x  100) 


20  percent 


Figure  5.  Sample  Method  3  (Step  4)  Problem  to  Induce  Expert's 
Indifference  Probability,  p. 


in  Figure  5.  The  expert  then  provides  a  second  response  to  an 
indifference  probability  problem  that  could  be  considered  the  inverse  of 
the  original.  The  expert's  response  to  the  first  indifference  probability 
problem  is  plotted  on  the  X-coordinate,  while  one  minus  the 
corresponding  response  from  the  second  indifference  probability  problem 
is  plotted  on  the  Y-coordinate.  Figure  6  demonstrates  an  example  of  a 
second  attribute  choice  indifference  probability  problem  corresponding 
to  the  first  problem  In  Figure  5. 


•You  currently  have  the  (intermediate)  $20.00  gift.  You  may  keep  it,  or  you 
may  use  it  to  gamble  for  the  (Best)  $100.00  gift.  If  you  gamble  and  win,  you 
get  the  (Best)  $100.00  gift.  If  you  gamble  and  loose,  you  get  the  (worst)  $5.00 
gift.  If  you  do  not  gamble  at  all,  you  keep  the  (intermediate)  $20.00  gift. 


•What  is  the  maximum  percentage  chance  of  getting  the  (worst)  $5.00  gift 
you  are  willing  to  accept  in  order  to  take  the  gamble? 


Expert's  Response,  (q  x  100)' 


70  percent 


Figure  6.  Sample  Method  3  (Step  4)  Problem  to  Induce  Second  Attribute 


Choice  Indifference  Probability,  q. 
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The  Indifference  probability  equates  the  expected  values  out 
of  either  branch  of  the  decision  node  in  the  decision  sapling  such  that  the 
expert  is  indifferent  between  taking  the  riskless  branch  or  the  risky  venture. 
When 

=  P^b  +  {1-P)Vw-  where  p  =  Pr{win},  (4) 

Vj  =  (l-q)V^  +  qV^  ,  where  q  =  Pr{loose},  and  (5) 

Vj, ,  Vj ,  Vv^  =  values  of  best,  intermediate  and  worst 
attribute  choices,  respectfully, 

the  expert  is  perfectly  consistent.  Figure  7  shows  a  consistency  plot  for  the 
responses  given  in  the  examples  in  Figures  5  and  6.  Note  that  if  the  expert 


Figure  7.  Method  3  (Step  4)  Indifference  Probability  Intra-Method 


Consistency  Check. 
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is  perfectly  consistent  with  his  responses,  then 


p  =  (1  -q). 


(6) 


2.  Inter-Method  Consistency  Checks 

Inter-method  consistency  check  procedures  are  presented  in  the 
following  subsections.  As  with  the  intra-method  consistency  checks,  these 
checks  involve  plotting  data  on  an  (X,  Y)  plot.  If  the  expert  is  perfectly 
consistent,  a  line  drawn  from  the  origin  through  the  (X,  Y)  data  point 
should  have  a  slope  of  1 .0. 

a.  Consistency  Check  Between  Methods  1  and  2 

The  first  inter-method  consistency  check  compares  the 
expert's  responses  to  Method  1  (marking  a  linear  scale)  and  Method  2 
(attribute  choice  pairwise  comparisons).  The  expert’s  Method  2  responses, 
the  r,^|'s,  are  plotted  on  the  X-coordinate.  The  corresponding  calculated 
ratios  of  values  from  Method  1  are  plotted  on  the  Y-coordinate.  For 
example,  from  Figure  2,  the  expert's  response  for  the  best  versus  worst 
pairwise  comparison  was  19.  From  Figure  1 ,  the  ratio  of  the  best  attribute 

Vb 

choice  value,  Vj^,  versus  the  worst  attribute  choice  value,  V^,  was  ^  = 
y  =  1 8.  If  the  expert  was  perfectly  consistent  between  methods  the 
plotted  point  should  lie  on  a  line  through  the  origin  of  slope  1 .0.  Figure  8 
shows  the  inter-method  consistency  check  for  the  data  from  the  figures 
mentioned  above. 

b.  Consistency  Check  Between  Methods  1  and  3 

A  method  for  checking  consistency  between  Methods  1 
and  3  is  now  discussed.  The  relationship  between  p  and  the  ratio  of  value 
differences  is  first  derived.  From  Equation  (4), 
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V,  =  pv,  +  (l-p)V^  , 

=  pVb  -  pVw  +  , 

(V|  -VJ  =  p(V,  -  Vj. 


Thus 


^  (Vj-Vw) 

P  (Vb-Vw)  • 


(7) 


Each  p  is  plotted  on  the  X-coordinate  and  the  corresponding  (Vb-^^' 
the  Y-coordinate.  Again,  if  the  expert  is  perfectly  consistent  between 
methods,  the  plotted  points  will  lie  directly  on  a  line  through  the  origin  of 
slope  1 .0. 
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Mean  Squared  Difference 

Sum  of  squared  data  point  vertical 
displacement  from  the  line,  divided  by  the 
number  of  data  points.  In  this  example,  mean 
squared  difference  is: 

[(4.0-3.5)^  +  (4.5-5.0)^  +  (18.0-19.0)^  ]  /  3  =  0.50, 

Vertical  Bias 

Sum  of  vertical  displacements  above  (+)  or 
below  (-)  the  line,  divided  by  the  number  of 
data  points.  In  this  example,  vertical  bias  is: 
(4.0-3.5)  +  (4.5-5.0)  +  (18.0-19.0)  =-1.0. 


Figure  8.  Inter-Method  Consistency  Check  between  Methods  1  and  2, 
Marking  a  Linear  Scale  versus  Attribute  Choice  Pairwise 
Comparisons. 
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Figure  9  shows  a  consistency  plot  of  the  Methods  1  and  3 
sample  data  from  Figures  1  and  5.  Note  that  since  probabilities  are  being 
plotted,  the  scales  on  this  consistency  plot  are  values  between  0.0  and 
1 .0,  unlike  the  previous  inter-method  consistency  plot  which  is  scaled  by 
ratios  of  attribute  values. 


Figure  9.  Inter-Method  Consistency  Check  between  Methods  1  and  3, 
Marking  a  Linear  Scale  versus  Attribute  Choice  Indifference 
Probabilities. 

c.  Consistency  Check  Between  Methods  2  and  3 

The  third  and  final  inter-method  consistency  check  compares 
the  expert's  responses  between  Methods  2  and  3.  From  Equation  (7), 
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The  value  of  p  is  plotted  on  the  X-coordinate,  and  the  ratio  on  the 
Y-coordinate.  A  consistency  plot  of  the  Methods  1  and  3  sample  data 
would  look  very  similar  to  Figure  9. 

This  chapter  provided  definitions  of  multi-attribute  decision 
alternative  sets,  attributes,  and  attribute  choices,  and  an  overview  of  how 
these  elements  are  combined  to  evaluate  decision  alternative  sets.  The 
employment  of  subjective  inputs  from  subject  matter  experts  was 
discussed,  as  were  three  methods  for  evaluating  attribute  choices.  A 
presentation  of  the  techniques  used  to  apply  the  methods,  along  with 
ways  to  conduct  intra  and  inter-method  consistency  checks,  concluded 
the  chapter. 

Chapter  III  goes  into  detail  on  the  experimental  approach 


utilized  for  this  thesis. 


III.  EXPERIMENTAL  APPROACH 


Chapter  III  highlights  the  experimental  approach  used  to  collect 
the  thesis  data.  Also  included  is  a  brief  discussion  of  the  history  and 
current  mission  applications  of  LAMPS.  The  importance  of  and  reasoning 
behind  presenting  the  pilots  with  a  standardized,  albeit  generalized, 
mission  are  also  discussed.  The  specific  attributes  and  attribute  choices 
used  for  the  experiment  are  then  presented,  along  with  samples  of  the  81 
possible  helicopter  system  suites  that  could  be  created.  The  chapter 
finishes  with  an  overview  of  the  survey  response  form  and  guide,  and  a 
discussion  on  the  actual  administration  of  the  surveys. 

A.  UNITED  STATES  NAVY  LAMPS  HELICOPTER  PILOT  SUBJECT  MAHER  EXPERT 
DECISION  MAKERS 

United  States  Navy  LAMPS  (Light  Airborne  Multi-Purpose  System)’ 
helicopter  pilots  were  selected  to  be  the  subject  matter  experts  (i.e., 
decision  makers)  for  an  experimental  application  of  the  methods 
described  in  Chapter  II  due  to  the  author's  extensive  experience  and 
familiarity  with  the  LAMPS  community.  A  total  of  27  LAMPS  pilots  assigned 
to  the  Naval  Postgraduate  School  were  individually  administered  a  survey 
to  solicit  their  expert  judgements  on  the  value  of  three  pieces  of 
helicopter  equipment  (attribute  choices)  in  each  of  four  different 
equipment  categories  (attributes).  The  survey  also  queried  the  subject 
matter  experts  to  solicit  data  for  developing  equipment  category 
weighting  factors  (attribute  weighting  factors)  for  possible  future  analysis. 

'  Both  U.  S.  Navy  LAMPS  Mk  I  and  LAMPS  Mk  III  helicopter  pilots  were  surveyed  for 
this  thesis.  LAMPS  Mk  I  pilots  operate  the  SH-2F/G  Seasprife,  while  LAMPS  Mk  III  pilots 
operate  the  SH-60B  Seahawk. 
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Data  processing  and  analysis  of  the  experts'  responses,  especially 
response  consistency  checks,  are  presented  in  Chapters  IV  and  V. 

The  reader  is  reminded  that  it  was  not  the  intent  of  this  thesis  to 
develop  values  or  weighting  factors  for  LAMPS  equipment,  systems,  or 
system  suites,  rather  it  was  to  exercise  and  check  for  expert  response 
consistency  of  the  methods  presented  in  Chapter  II. 

B.  WHAT  IS  LAMPS? 

LAMPS'  ancestor  was  conceived  in  the  late  1 950's  in  the  form  of 
DASH  (Drone  Anti-Submarine  Helicopter).  DASH  was  an  unmanned,  radio 
controlled,  torpedo  carrying  helicopter  designed  to  deliver  USW 
(UnderSea  Warfare)  weapons  at  convergence  zone  ranges,  well  beyond 
the  five  mile  range  of  the  parent  ship.  Deployed  from  USW  ships,  DASH 
was  originally  semi-  expendable,  which  is  just  as  well  as  few  DASH's  lived 
beyond  a  few  flights. 

As  torpedo  size  increased,  so  did  the  payload  requirements,  size 
and  costs  of  DASH.  By  the  late  1960's,  DASH's  costs  no  longer  justified 
expendability.  By  this  time,  helicopter  and  powerplant  technology  had 
matured  to  the  point  where  manned  helicopters  could  be  deployed  from 
surface  combatants.  In  1 971 ,  LAMPS  Mk  I  was  born  in  the  form  of  the 
SH-2F  Seasprite.  In  its  infancy,  LAMPS  Mk  I  was  a  marriage  of  off-the-shelf 
sensor  and  avionic  systems  with  previously  boneyarded  airframes.  It  was 
an  enormously  successful  relationship  and  the  Navy  quickly  recognized 
the  synergistically  enhanced  capabilities  of  a  ship-helicopter  system. 

In  1975,  IBM  won  the  System  Integration  Program  contract  to  design 
LAMPS  Mk  III  from  the  ground  up.  The  LAMPS  Mk  III  SH-60B  Seahawk  first 
deployed  in  1 984  and  is  the  airborne  component  of  a  fully-integrated 
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ship-helicopter  weapon  system.  The  helicopter  is  crewed  by  a  pilot,  ATO 
(Airborne  Tactics  Officer),  and  an  enlisted  sensor  operator  (AW  rating).  It  is 
highly  capable  of  autonomous  operations  with  all  of  its  sensors  and 
weapons,  including  radar,  sonobuoys,  MAD  (Magnetic  Anomaly 
Detector),  IFF  (Identification  Friend  or  Foe),  ESM  (Electronic  Support 
Measures),  torpedoes,  and  Penguin  Missiles. 

The  helicopter  is  only  half  the  story,  however,  as  many  of  the 
airborne  sensors'  capabilities  are  greatly  enhanced  through  a  dedicated 
datalink  to  a  LAMPS  Team  embarked  in  the  parent  ship.  The  ship's  greater 
data  processing  and  analysis  capabilities,  especially  for  sonar  and  ESM 
data,  along  with  greater  C^l  (Command,  Control,  Communications  and 
Intelligence)  capability,  make  for  a  truly  synergistic  system. 

LAMPS  was  first  cast  primarily  as  an  USW  platform.  Today  LAMPS' 
two  primary  missions  are  ASUW  (Anti-SUrface  Warfare)  and  USW.  The  ever¬ 
growing  list  of  secondary  missions  is  expansive,  and  includes  ASST 
(Anti-Ship  Surveillance  and  Targeting),  NSFS  (Naval  Surface  Fire  Support), 
SAR  (Search  And  Rescue),  VERTREP  (VERTical  REPIenishment),  and 
MEDEVAC  (MEDical  EVACuation),  to  name  a  few. 

Currently  the  LAMPS  Mk  I  is  being  phased  out  of  the  fleet,  while  the 
Mk  III  is  deployed  extensively,  embarked  in  Ticonderoga  Class  Cruisers 
(CG-47),  Spruance  (DD-963)  and  future  Arleigh  Burke  (DDG-51)  Class 
Destroyers,  and  Oliver  Hazzard  Perry  Class  Frigates  (FFG-7).  [Refs.  2-6] 

C.  STANDARDIZED  MISSION 

In  view  of  the  wide  range  and  variety  of  missions  performed  by 
LAMPS  aircrews,  and  the  numerous  significant  operational  and  tactical 
considerations  that  dictate  which  helicopter  systems  to  use  and  how  to 
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employ  them  given  a  specific  mission,  a  standardized  mission  was 
presented  to  the  LAMPS  pilots  from  which  to  reference  their  responses. 

The  standardized  mission  presented  to  the  LAMPS  pilots  was  a 
generic  USW  mission^.  Before  taking  the  survey,  piiots  were  briefed  to  taiior 
their  responses  in  reference  to  a  complete  USW  mission;  i.e.,  from  the 
redetection  phase  all  the  way  through  to  completion  of  the  battle 
damage  assessment  (BDA)  phase.  Pilots  were  repeatedly  reminded 
throughout  the  survey  to  reference  their  responses  to  a  complete  USW 
mission.  Pilot  response  in  reference  to  a  generalized  USW  mission  was 
selected  over  presenting  a  more  detailed  mission  scenario  to  avoid 
possible  complications  caused  by  information  overload.  It  was  felt  that 
giving  further  mission  scenario  details  (e.g.,  target  type  (nuclear  or  diesel 
submarine),  weather  conditions,  water  characteristics  (depth, 
temperature  gradients,  salinity,  biologies,  etc.),  area  (littoral  or  blue 
water),  etc.),  would  generate  more  confusion  than  ciarification. 

D.  EXPERIMENT  AHRIBUTES,  AHRIBUTE  CHOICES,  AND  MULTI-AHRIBUTE 
DECISION  ALTERNATIVE  SETS 

Four  LAMPS  heiicopter  system  categories  vital  to  the  successful 
execution  of  a  complete  USW  mission  were  presented  to  the  pilots.  The 
four  system  categories,  weapons,  navigation  systems,  communication 
systems,  and  sensors,  made  up  the  attributes  for  the  methods  described  in 
Chapter  II.  Each  of  these  attributes  was  comprised  of  three  different 
system  options  (i.e.,  the  attribute  choices).  Table  1  lists  the  attributes  and 
corresponding  attribute  choices  used  in  the  survey. 

^  USW  is  a  relatively  new  term  to  represent  what  historically  has  been  called 
Anti-Submarine  Warfare  or  ASW. 
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Attr 

ibute,  i  1 

Weapons, 

1 

Navigation 

Systems, 

2 

Communication 

Systems, 

3 

Sensors, 

4 

Attribute 

Choices 

(abbreviation) 

(Note,  no 
ranking 
implied  by 
table  ordering 
of  attribute 
choices.) 

Mk-46 

Torpedo 

(Mk-46) 

Global 

Positioning  System 
(GPS) 

Datalink 

(D/L) 

Magnetic 
Anomaly 
Detector  (MAD) 

500#  Depth 
Charge 
(500#) 

Tactical  Air 
Navigation 
(TACAN) 

Satellite 

Communications 
(Sat  Com) 

Surface 
Search  Radar 
(Radar) 

Mk-50 

Torpedo 

(Mk-50) 

Doppier 

Radar 

(Dop) 

UHF/VHF 

Radio 

(Radio) 

Forward  Looking 
Infrared 
(FLIR) 

Table  1 .  Survey  Attributes  and  Attribute  Choices. 


Given  the  LAMPS  system  categories  and  individual  systems  (i.e.,  the 
attributes  and  attribute  choices),  LAMPS  helicopter  USW  mission  system 
suites  (i.e.,  alternative  sets)  could  be  formed.  Any  system  suite  would  be 
comprised  of  one  individual  system  option  from  each  of  the  four  system 
categories.  Given  four  attributes  and  three  attribute  choices  per 
category,  there  would  be  j  =  81  ( S'*  =  81  )  possible  LAMPS  helicopter 
mission  system  suites  for  the  pilots  to  rank.  For  example: 
j=  1;  (Mk-46,  GPS,  D/L,  MAD), 

]  =  2;  (Mk-46,  GPS,  D/L,  Radar), 
j  =  3;  (Mk-46,  GPS,  D/L,  FUR), 
j  =  4;  (Mk-46,  GPS,  Sat  Com,  MAD), 


j  =  81:(Mk-50,  Dop,  Radio,  FLIR). 
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E.  DATA  COLLECTION  SURVEY 

Copies  of  the  survey  instruction  guide  and  survey  response  form 
used  to  collect  data  are  in  Appendix  B.  Each  survey  was  individually 
administered  with  the  author  readily  available  to  answer  questions 
regarding  the  mechanics  or  semantics  of  the  survey.  The  author  made 
every  effort  to  not  plant  response  predispositions  in  pilots  taking  the 
survey.  Pilots  were  not  told  until  after  completing  the  survey  that  their 
responses  would  be  examined  primarily  for  consistency  as  opposed  to 
trying  to  quantify  the  value  of  a  LAMPS  system  suite.  Most  of  the  27  pilots 
taking  the  survey  completed  it  in  45  +  1 0  minutes.  With  few  exceptions 
the  survey  follows  the  order  of  steps  described  in  Chapter  II. 

This  chapter  presented  the  experimental  approach  used  to  collect 
data  for  the  thesis,  as  well  as  brief  discussions  of  the  history  and  current 
mission  applications  of  LAMPS.  The  importance  of  and  reasoning  behind 
presenting  the  pilots  with  a  standardized  mission  were  also  discussed.  The 
specific  attributes  and  attribute  choices  used  for  the  experiment  were 
presented,  along  with  samples  of  the  81  possible  helicopter  system  suites 
that  could  be  created.  The  chapter  closed  with  an  overview  of  the 
survey  response  form  and  guide,  and  a  discussion  on  the  actual 
administration  of  the  surveys. 

Chapter  IV  presents  the  methods  used  to  analyze  the  data, 
specifically  infra  and  inter-method  consistency  checks. 
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IV.  CONSISTENCY  WITHIN  AND  BETWEEN  METHODS 

This  chapter  describes  the  methods  used  to  andlyze  the  data, 
specifically  intra  and  inter-method  consistency  checks.  Consistency 
check  plots,  along  with  quantitative  consistency  measures,  are  presented 
for  five  main  data  subsets. 

A.  CONSISTENCY  CHECKS  OVERVIEW 

Five  sets  of  consistency  checks  are  discussed  in  this  chapter,  two 
intra-method  and  three  inter-method.  Each  set  of  checks  employs  three 
techniques  for  analyzing  the  LAMPS  pilots'  survey  response  consistencies. 
The  first  technique  involves  plotting  corresponding  data  subsets  on  an 
(X,  Y)  plot  along  with  a  line  through  the  origin  of  slope  1 .0.  If  the 
experimental  responses  are  perfectly  consistent,  all  the  data  points  lie  on 
the  line.  The  reader  can  see  in  Figures  10  through  14  in  the  subsequent 
sections  of  this  chapter  that  while  some  of  the  plots  visually  indicate  a 
moderate  degree  of  consistency,  others  appear  to  show  a  great  deal  of 
inconsistency. 

The  second  consistency  check  technique  examines  the  square  root 
of  the  mean  squared  difference,  or  dispersion  parameter,  Vd^  =  d ,  of  the 
data  points'  vertical  displacement  from  a  line  through  the  origin  of  slope 
1 .0.  The  magnitude  of  d  gives  a  measure  of  the  degree  of  consistency. 
The  value  of  d  will  be  zero  if  and  only  if  all  responses  are  perfectly 
consistent.  Otherwise  d  will  have  a  positive  value.  Equation  (9)  gives  the 
formulation  for  the  mean  squared  difference,  d^: 

d2  =  ^Ek=i(yk  -Xk)^  ,  where  (9) 
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=  mean  squared  difference, 

K  =  the  number  of  data  points  in  the  subset, 

X|^  =  the  X-coordinate  of  the  k*'"  data  point,  and 
=  the  Y-coordinate  of  the  data  point. 


Suppose  the  X-coordinate  is  plotted  over  a  range  (0,  m),  so  a 
consistent  point  vy/ith  x  =  m  would  be  (m,  m).  The  scale  can  be  converted 
to  (0, 1 )  by  dividing  each  data  point  by  m,  so  if  d^^^^  is  defined  to  be  the 
normalized  mean  squared  difference,  from  Equation  (9), 


-  iy'^  ('h.  ^ 

-  K  ■^k=l  Vm  mt  ~  „2  ' 


and  the  normalized  dispersion  parameter  is 


The  units  for  the  graphs  shown  in  Figures  11, 13,  and  14  are 
indifference  probabilities  and  range  in  magnitude  from  zero  to  one.  The 
units  for  the  graphs  shown  in  Figures  10  and  12  are  ratios  of  attribute 
choice  values.  The  X-coordinate  axes  for  these  graphs  are  limited  to  10.0 
and  the  Y-coordinate  axes  to  20.0.  In  order  to  make  quantitative 
comparative  statements  between  consistency  checks  of  different  units, 
the  d’s  for  the  checks  involving  attribute  choice  value  ratios  were 
normalized  to  a  (0, 1 )  scale.  This  was  done  by  dividing  d  by  m  =  1 0,  the 
maximum  value  of  the  X-coordinate  axis.  After  normalizing  the  ratio  unit 
data  to  the  same  scale  as  the  probability  unit  data,  the  d„„Js  from  all  the 
data  subsets  can  be  compared  directly. 
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The  final  technique  used  to  examine  response  consistency 
calculates  the  normalized  vertical  bias,  from  a  line  through  the  origin 
of  slope  1 .0.  The  sign  of  b  indicates  to  which  side  of  the  slope  1 .0  line  the 
data  is  biased.  A  positive  b  indicates  bias  above  the  line,  negative 
indicates  below.  The  magnitude  of  b^^^^  indicates  the  degree  to  which 
the  data  set  is  biased.  Equation  (11)  gives  the  formulation  for  the 
normalized  vertical  bias,  b _ : 

norm 

bnorm  =  ^  2^  (Y k  "  Xk)  ,  Where  (11) 

-  normalized  vertical  bias, 
m  =  maximum  value  for  the  X-axis, 

K  =  the  number  of  data  points  in  the  subset, 

x, ^  =  the  X-coordinate  of  the  data  point,  and 

y, ^  =  the  Y-coordinate  of  the  data  point. 

The  value  of  m  is  1 .0  for  the  data  subsets  with  indifference  probability  units, 
and  10.0  for  the  data  subsets  with  attribute  choice  value  ratio  units. 

Appendix  C  is  a  tabulation  of  the  raw  survey  data.  Supplementary 
graphs  of  numerous  data  subsets  are  in  Appendix  D. 

B.  EXPERIMENTAL  INTRA-METHOD  CONSISTENCY  CHECKS 

The  pilots’  responses  to  Methods  2  and  3,  attribute  choice  pairwise 
comparisons  and  indifference  probabilities,  were  examined  for 
intra-method  consistency. 

1.  Method  2:  Attribute  Choice  Pairwise  Comparisons 

Figure  10  presents  a  graph  plotting  the  best  versus  worst  pairwise 
comparison  ratio,  r^,^  ,  directly  from  the  data  as  the  X-coordinate,  and  the 
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"Best  vs  Worst"  Pairwise  Comparison  Ratio,  rbw 
Weapons^  Navigation,  Communications,  Sensors 


Figure  10.  Attribute  Choice  Pairwise  Comparison  intra-Method 
Consistency  Check,  all  Attributes. 


product  of  the  best  versus  intermediate  ratio  times  the  intermediate 
versus  worst  ratio,  r^.  r.^.  as  the  corresponding  Y-coordinate.  This  graph 
presents  all  the  attribute  choice  pairwise  comparison  data,  within  the  limits 
of  the  graph,  from  all  four  attributes  (weapons,  navigation  systems, 
communication  systems,  and  sensors).  Visual  inspection  of  the  graph 
reveals  the  derived  best  versus  worst  ratios,  the  Y-coordinates,  are 
somewhat  consistent  with  the  ratios  obtained  directly,  and  are  definitely 
biased  towards  the  derived  ratios.  The  plotting  method  used  does  not 


32 


indicate  multiple  points  In  the  same  location,  so  It  is  difficult  to  get  a  true 
visual  picture  of  the  amount  of  consistency  or  bias.  The  values  of  d^^^^ 
and  are  0.0299  and  +0.1401 ,  respectively,  indicating  a  high  level  of 
consistency,  but  a  moderate  level  of  bias  in  the  answers.  Supplementary 
graphs  displaying  the  data  for  each  of  the  four  attributes  are  found  in 
Appendix  D. 

2.  Method  3:  Attribute  Choice  indifference  Probabiiities 

Figure  1 1  presents  a  plot  of  the  indifference  probability,  p,  directly 
from  the  data  on  the  X-coordinate,  and  the  corresponding  one  minus  the 
second  Indifference  probability,  q,  as  the  Y-coordinate.  This  graph 
presents  all  the  attribute  choice  indifference  probability  data,  both  first 


Indifference  Probability,  p 
Weapons,  Navigation,  Communications,  Sensors 


Figure  1 1 .  Attribute  Choice  Indifference  Probabilities  Intra-Method 
Consistency  Check,  all  Attributes. 
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second,  from  all  four  attributes.  Visual  inspection  of  the  graph  indicates  a 
less  consistent  set  of  responses  than  with  the  pairwise  comparison  data  of 
the  previous  graph,  but  a  much  smaller  positive  bias  as  the  data  is  much 
more  evenly  spread  both  above  and  below  the  line.  For  these 
observations  d^g^^  =  0.1851  andb^^rm"  +0.0420.  Supplementary  graphs 
displaying  the  data  for  each  of  the  four  attributes  are  found  in 
Appendix  D. 

C.  EXPERIMENTAL  INTER-METHOD  CONSISTENCY  CHECKS 

The  pilots'  responses  to  all  three  methods,  marking  a  linear  scale, 
attribute  choice  pairwise  comparisons,  and  indifference  probabilities, 

were  examined  for  inter-method  consistency. 

1 .  Between  Methods  1  and  2:  Marking  a  Linear  Scale  versus 
Attribute  Choice  Pairwise  Comparisons 

Figure  1 2  presents  a  plot  of  pairwise  comparison  data  on  the 
X-coordinate,  and  corresponding  ratios  of  linear  scale  values  from 
Method  2  on  the  Y-coordinate  (see  Equation  (3)).  The  degree  of 
consistency  between  Methods  1  and  2  at  first  appears  to  be  notably  less 
than  the  degree  of  consistency  within  either  Methods  1  or  2.  A  second 
characteristic  difference  from  the  intra-method  checks  is  the  vertical 
grouping  of  the  data  points.  Intuitively  this  is  explained  by  the  pilots 
propensity  for  integer  values  when  providing  ratios  of  attribute  choice 
vaiues  inputs.  Over  91  percent  of  the  points  on  the  X-coordinate  are 
integer  value. 

The  value  of  d  for  this  data  is  only  0.0361 ,  almost  the  same  as 

1 1  »  ■^1  ww  norm 

the  best  intra-method  value.  This  result  seems  to  contradict  what  is 
displayed  graphically,  until  the  data  is  examined  more  closely.  Over  79 
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percent  of  the  X-coordinate  values  are  five  or  less,  and  nearly  40  percent 
are  two  or  less  (out  of  286  data  points).  The  relatively  small  value  for  d^^^ 
is  then  somewhat  explained  by  the  much  larger  volume  of  data  points  in 
the  groupings  In  the  lower  left  corner  of  the  graph  and  close  to  the 
slope  1 .0  line. 

Finally,  and  not  surprisingly  looking  at  the  plot,  this  check  was 
strongly  negatively  biased  with  a  b^^^^  of  (-0.2264).  Supplementary 
graphs  displaying  the  data  for  each  of  the  four  attributes,  and  for  the 
three  types  of  attribute  choice  pairwise  comparisons,  are  found  in 
Appendix  D. 


Figure  12.  Marking  a  Linear  Scale  versus  Attribute  Choice  Pairwise 
Comparisons  Inter-Method  Consistency  Check,  all  Attributes. 
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2.  Between  Methods  1  and  3:  Marking  a  Linear  Scaie  versus 
Attribute  Choice  Indifference  Probabilities 

Figure  13  presents  a  plot  of  the  indifference  probabilities  directly 
obtained  from  Method  3  on  the  X-coordinate.  The  corresponding  derived 
indifference  probabiiities,  from  Method  1  data,  is  plotted  on  the 
Y-coordinate  (see  Equation  (7)).  This  graph  also  shows  some  pilot 
disposition  for  vertical  grouping  of  data  points,  though  not  quite  as  much 
as  in  the  previous  graph.  The  raw  data  for  the  X-coordinates  are 
percentages.  Only  three  of  the  108  X-coordinate  inputs  are  integer 


Figure  13.  Marking  a  Linear  Scale  versus  Attribute  Choice  Indifference 
Probabilities  Inter-Method  Consistency  Check,  ail  Attributes. 
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values  not  divisible  by  five,  and  only  30  of  the  108  are  not  divisible  by  ten. 
Reviewing  the  data  helps  explain  why  the  vast  majority  of  data  points  fall 
on  the  vertical  gridlines. 

The  value  of  d^^^  for  this  check  is  0.2968,  significantly  greater  than 
the  three  previous  consistency  check  results.  Reviewing  the  graph 
confirms  the  large  d^^^  value,  seemingly  indicating  a  relatively  low  level 
of  inter-method  consistency  between  Methods  1  and  3. 

Vertical  bias  is  relatively  small,  as  might  be  expected  by  the 
considerable  and  apparently  evenly  distributed  vertical  displacement  of 
the  data  points  on  either  side  of  the  slope  1 .0  line.  The  value  of  b^^^^  is 
(-0.0862),  the  least  biased  of  the  inter-method  consistency  checks 
discussed  in  this  chapter.  Suppiementary  graphs  displaying  the  data  for 
each  of  the  four  attributes  are  in  Appendix  D. 

3.  Between  Methods  2  and  3:  Attribute  Choice  Pairwise 

Comparisons  versus  Attribute  Choice  indifference  Probabilities 

The  final  inter-method  consistency  check  compares  directly 
obtained  indifference  probabilities  and  corresponding  indifference 
probabiiities  derived  from  the  attribute  choice  pairwise  comparison 
method  data  (see  Equation  (8)).  Figure  14  presents  a  plot  of  the 
directly-obtained  indifference  probabilities  on  the  X-coordinate  and  the 
corresponding  derived  values  on  the  Y-coordinate.  Since  this  graph  has 
the  same  inputs  for  the  X-coordinates  as  the  previous  graph,  it  aiso  shows 
correlation  along  the  vertical  gridlines.  The  graph  indicates  a  rather  low 
level  of  consistency  and  a  strongly  negative  bias. 

The  value  of  d^^^  for  this  check  is  0.3962,  the  largest  d„^^  value  of 
the  five  consistency  checks  discussed  in  this  chapter,  and  greater  than  all 
but  one  of  the  data  subset  checks  found  in  Appendix  D.  Such  a  large 
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value  of  suggests  a  significant  level  of  response  inconsistency 
between  Methods  2  and  3. 

The  graphical  interpretation  of  the  bias  is  supported  by  a  b^^^  of 
(-0.2052),  Indicating  a  strongly  negative  bias.  Supplementary  graphs 
displaying  the  data  for  each  of  the  four  attributes  are  found  in 
Appendix  D. 


Figure  1 4.  Inter-Method  Consistency  Check,  Attribute  Choice  Pairwise 
Comparisons  versus  Indifference  Probabilities,  all  Attributes. 
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D.  SUMMARY  OF  CONSISTENCY  CHECK  RESULTS 


Table  2  summarizes  the  d,  and  number  of  data  points, 

K,  for  each  of  the  consistency  checks  discussed  in  this  chapter.  Table  3 
summarizes  the  same  information  for  the  data  subset  consistency  check 
graphs  found  in  Appendix  D. 


Consistency  Check  (Figure  Number) 

Intra-method  Consistency  Checks 

d* 

d 

D 

Method  2: 

Pairwise  Comparisons  (lO) 

Entire  data  set  (w/in  graph  limits),  all  attributes 

8.9314 

2.9885 

0.0299 

+0.1401 

71 

Method  3; 

Indifference  Probabilities  (ii) 

Entire  data  set,  all  attributes 

0.0343 

0.1851 

0.1851 

+0.0420 

108 

■ 

Inter- method  Consistency  Checks 

d* 

d 

BHI 

D 

Methods  1  and  2: 

Marking  a  Linear  Scale  vs  Pairwise 

Comparisons  (12) 

Entire  data  set  (w/in  graph  limits),  all  attributes 

12.968 

3.601 

0.0361 

-0.2264 

286 

Methods  1  and  3: 

Marking  a  Linear  Scale  vs  Indifference 
Probabilities  (13) 

Entire  data  set,  all  attributes 

0.0881 

0.2968 

0.2968 

-0.0862 

108 

Methods  2  and  3: 

Pairwise  Comparisons  vs  Indifference 
Probabilities  (14) 

Entire  data  set,  all  attributes 

0.1569 

0.3962 

0.3962 

-0.2052 

103 

Table  2.  Summary  Table  of  Consistency  Check  Parameters  for  Data 


Subsets  Presented  in  Chapter  IV. 
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8.6406  2.9395  0.0294 


5.4167  2.3274  0.0233 


5.5559  2.3571 


13.216  3.6354  0.0364 


0.0245 

0.1566 

0.0484 

0.2201 

0.0154 

0.1240 

0.0487 

0.2207 

Consistency  Check  (Figure  Number) 


Intra-method  Consistency  Checks 


Method  2:  Pairwise  Comparisons 

Weapons  (a.d-1) 


Navigation  (a.d-2) 


Communications  (a.d-3) 


Sensors  (a.d-4) 


Method  3:  indifference  Probabilifies 

Weapons  (a.d-5) 


Navigation  (a.d-6) 


Communications  (a.d-7) 


Sensors  (a.d-8) 


Inter-method  Consistency  Checks 


Methods  1  and  2: 

Marking  Linear  Scaie  vs  Pairwise  Comparisons 

Best  vs  Intermediate  {a.d-9) 


Best  vs  Worst  (a.d-io) 


intermediate  vs  Worst  (a.d-ii) 


Weapons  (a.d-12) 


Navigation  (A.d-13) 


Communications  (a.d-M) 


Sensors  (a.d-is) 


Methods  1  and  3:  Marking  Linear  Scale  vs 

indifference  Probabilifies 

Weapons  (a.d-16) 


Navigation  (a.d-17) 


Communications  (a.d-18) 


Sensors  (a.d-19) 


Methods  2  and  3:  Pairwise  Comparisons  vs 
Indifference  Probabilifies 

Weapons  (a.d-20) 


Navigation  (a.d-21) 


Communications  (a.d-22) 


Sensors  (a.d-23) 


Table  3.  Summary  Table  of  Consistency  Check  Parameters 
Subsets  Presented  in  Appendix  D. 


+0.1538  .  20 


+0.1250  12 


+0.0704  27 


13.077  3.6162  0.0362  -0.2393  103 


-0.3008  80 


-0.3460  65 


-0.2376  69 


-0.1738  76 


-0.1556  27 


18.641  4.3176  0.0432 


8.0895 


9.5271 


22.787  4.7736  0.0477 


13.341  3.6526  0.0365 


7.6704  2.7696  0.0277 


0.0759  0.2754  0.2754 


0.0995  0.3155  0.3155 


0.0630  0.2509  0.2509 
0.1139  0.3375  0.3375 


0.1316  0.3627  0.3627 


0.2103  0.4586  0.4586 
0.1 285 1  0.3587  0.3585 
0.1689  0.1689 


-0.1513  27 


-0.0299  27 

-0.0078  27 


-0.2761  26 


-0.2976  27 

-0.1 305 1  25 
-0.09 


for  Data 
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This  chapter  described  the  methods  used  to  analyze  the  data, 
specifically  intra  and  inter-method  consistency  checks.  Consistency 
check  plots  and  quantitative  measures  of  consistency  were  presented  for 
five  main  data  subsets.  The  chapter  finished  with  tables  summarizing  the 
consistency  check  parameters  for  28  various  data  subsets. 

The  final  chapter  will  interpret  the  consistency  check  parameters 
developed  in  Chapter  IV  for  the  five  main  data  subsets. 
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‘  V.  SUMMARY  AND  CONCLUSIONS 

This  chapter  provides  interpretations  of  the  consistency  check 
parameters  developed  in  Chapter  IV  for  the  five  main  data  subsets. 

Some  consistency  checks  that  yielded  unexpected  results  are  analyzed  in 
greater  detail  and  utility  function  theory  applied  in  an  attempt  to  explain 
the  results.  The  thesis  closes  with  discussions  on  applicability  of  the 
methods,  experiment  conclusions  and  suggested  areas  for  future  study. 

A.  EXPERIMENT  RESULTS 

The  following  is  a  discussion  of  the  experiment  results.  Table  4  Is  a 
reduced  version  of  Table  2  and  presents  the  primary  consistency  check 
parameters,  the  normalized  dispersion  parameter,  d^^^^ ,  and  the 
normalized  bias,  b„„,^,  for  the  two  intra-method  and  three  inter-method 

norm 

consistency  checks  encompassing  all  attributes. 

1.  Intra-method  Consistency  Checks 
a.  Within  Method  2 

The  dispersion  consistency  check  parameter,  d^^^^ ,  for 
Method  2  was  only  0.0299,  indicating  a  low  level  of  data  spread  within  this 
method. 

Expert  responses  to  this  method  were  moderately  positively 
biased,  with  b  =  +0.1 401 .  From  the  discussion  in  Chapter  II,  Subsection 

norm 

D.l  .c,  and  Equation  (3),  we  know  that  if  the  experts  were  perfectly 
consistent,  then 


rbw  —  rbirjw 
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Consistency  Check  (Figure  Number) 

1 

Intra-method  Consistency  Checks 

Method  2: 

Pairwise  Comparisons  (lo) 

Entire  data  set  (w/in  graph  limits),  all  attributes 

0.0299 

+0.1401 

Method  3: 

Indifference  Probabilities  (ii) 

Entire  data  set,  all  attributes 

0.1851 

+0.0420 

Inter-method  Consistency  Checks 

norm 

^norm 

Methods  1  and  2: 

Marking  a  Linear  Scale  vs  Pairwise  Comparisons  (i2) 
Entire  data  set  (w/in  graph  limits),  all  attributes 

0.0361 

-0.2264 

Methods  1  and  3: 

Marking  a  Linear  Scale  vs  Indifference  Probabilities  (i3) 
Entire  data  set,  all  attributes 

0.2968 

-0.0862 

Methods  2  and  3: 

Pairwise  Comparisons  vs  Indifference  Probabilities  (U) 
Entire  data  set,  all  attributes 

0.3962 

-0.2052 

Table  4.  Summary  of  Primary  Consistency  Check  Parameters. 


A  positive  bias  indicates  that,  on  average,  experts  overestimated  the 
derived  values  for  r^,^  relative  to  the  directly  obtained  values. 

b.  Within  Method  3 

Consistency  check  parameter  magnitudes  for  Method  3 
were  the  converse  of  those  for  Method  2.  The  d^^^  value  was  relatively 
large  at  0.1851 ,  while  a  nearly  negligible  level  of  bias  was  indicated  with  a 
b^orm  of  +0.0420.  Figure  1 5  replots  the  data  for  Method  3  with  p 
plotted  on  the  horizontal  axis  and  p  -  (1  -  q)  plotted  on  the  vertical  axis. 

For  this  plot,  the  horizontal  axis  effectively  displays  the  line  of  slope  1 .0 
from  Figure  1 1 .  This  presentation  confirms  the  consistency  parameters 
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Figure  15.  Alternate  Intra-Method  3  Check:  p  -  (1  -  q)  versus  p. 

analysis  that  the  derived  values  for  p,  the  (1  -  q)'s,  are  rather  widely,  but 
fairly  evenly  dispersed  on  either  side  of  p. 

Figure  16  pictorially  demonstrates  Equation  (6),  i.e.,  the  two 
decision  saplings  are  equivalent  if  and  only  if 

p  =  (1  -  q),  and  q  =  (1  -  p),  where  0  <  p,  q  <  1 . 


Figure  1 6.  Two  Equivalent  Decision  Saplings. 


The  parameter  values  indicate  the  experts  may  have  had 
difficulty  translating  a  corresponding  value  of  q  for  a  given  p.  This 
conclusion  is  supported  by  the  fact  that  survey  steps  4(a)  and  (b)  (see 
Appendix  B)  used  to  collect  data  inputs  for  p  and  q,  generated  the 
largest  number  of  clarification  requests  from  the  experts  while  they  were 
taking  the  survey.  A  supporting  interpretation  of  these  results  is  that  while 
marking  a  linear  scale  or  conducting  pairwise  comparisons  were  rather 
natural  actions  for  the  experts,  the  experts  were  not  practiced  nor  familiar 
with  the  concept  of  indifference  probabilities  as  a  tool  for  making 
comparisons.  Expert  training  and  practice  with  indifference  probabilities 
might  very  well  reduce  the  response  dispersion  for  this  method,  improving 
the  level  of  consistency. 

2.  Inter-method  Consistency  Checks 
a.  Between  Methods  1  and  2 

The  d  „  „  value  for  the  check  between  Methods  1  and  2  was 
only  0.0361 ,  indicating  a  relatively  high  level  of  response  consistency 
between  these  two  methods.  The  inter-method  bias,  however,  was 
substantial  and  strongly  negative,  with  b^^,^  =  (-0.2264).  A  negative  bias 
indicates  that,  on  average,  the  derived  attribute  value  ratios  were 
understated  relative  to  the  directly  obtained  ratios.  The  relatively  large 
magnitude  of  the  bias  suggests  the  possibility  of  a  systematic  prejudice  in 
the  method,  possibly  induced  by  the  linear  scale  presentation.  Reviewing 
Equation  (3),  if  the  consistency  relationship  is  negatively  biased,  then,  on 
average 


Tbi 


Vb 

Vi 


r 

'bw  > 


and 


Vi 
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Given  the  above,  If  a  constant  inter-Method  1  and  2 


correction  factor,  c,2 ,  such  that  0  <  c,2  <  is  subtracted  from 

and  y^^^^rninoior '  ^ho  ratio  )  increases  in 

magnitude.  Thus,  there  is  a  value  of  c,2  for  each  data  point  such  that 
when  subtracted  from  the  values  in  both  the  numerator  and 
denominator,  the  ratio  of  values  equals  the  directly  obtained  ratio.  The 
effect,  then,  of  c,2  for  a  negatively  biased  relationship  between  Methods 
1  and  2,  is  to  effectively  shift  the  origin  of  the  linear  scale  to  the  right.  For 
example,  if  r^^  =  5,  =  20  and  =  60,  then 

rbw  =  5>f  =  3  =  ^  ,and 


r  —  R  — 

•bw  -  O-  v^_C,2  -  20-C,2 


Ci2  =  10  . 


Figure  1 7  pictorially  demonstrates  the  effect  of  the  c,2  =  10  for 
the  r^,^,  Vjj,  and  in  the  example.  The  strong  negative  bias  suggests  the 
experts  had  a  propensity  for  the  high  end  of  the  linear  scale,  and/or  an 
aversion  to  the  low  end. 


Figure  17.  Pictorial  Representation  of  Correction  Factor  c, 2. 


If  b  >0,  then  a  constant  c,,  is  added  to  both  the 
numerator  and  denominator.  This  effectively  shifts  the  origin  of  the  linear 
scale  to  the  left.  Application  of  a  correction  factor  to  the  entire  data 
subset  might  therefore  be  used  to  reduce  systematic  bias  in  the  experts' 
responses  due  to  a  propensity  for  either  end  of  the  linear  scale. 

Finding  a  globally  optimal  value  of  c,2  directly  is  difficult 
because  of  the  existence  of  local  minima  and  maxima.  Heuristic  and 
iterative  approaches  to  find  a  near  optimal  0,3,  however,  are  relatively 
easy  to  perform.  To  test  the  concept  of  a  correction  factor,  integer 
valued  from  one  to  ten  were  iteratively  applied  to  the  experimentai 
data  and  the  effects  on  b„„,^  noted.  The  magnitude  of  b,„^  was 
reduced  from  an  uncorrected  value  of  (-0.2264),  to  a  negligible  value  of 
0.0241  when  c,2  =  9,  a  tenfold  improvement. 

b.  Between  Methods  1  and  3 

The  value  of  d^^^  for  the  check  between  Methods  1  and  3 
was  0.2968,  which  seems  to  indicate  a  low  level  of  response  consistency 
between  these  two  methods.  The  modest  b^^^  of  (-0.0862)  was  the 
smallest  level  of  bias  of  the  three  inter-method  consistency  checks.  To 
investigate  the  high  value  of  d^^^^  further.  Figure  1 8  repiots  the  data  for  this 
inter-method  check  with  p  plotted  on  horizontal  axis,  and 
plotted  on  the  vertical  axis.  As  with  the  Method  3  intra-method 
consistency  check,  this  presentation  confirms  the  consistency  parameters 
analysis  that  the  derived  values  for  p  are  widely,  but  fairly  evenly 
dispersed  on  either  side  of  p.  An  interesting  and  unanticipated 
phenomenon  also  surfaced  in  Figure  18.  The  bias,  which  is  the  derived 
minus  the  directly  obtained  indifference  probability,  appears  to  become 
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more  negative  with  increasing  p.  This  data  behavior  warranted  closer 
examination  to  search  for  an  explanation. 


Figure  18.  Alternate  Inter-Methods  1  and  3  Check: 
P  -  [(V,  -  VJ/(V^,  -  V^)]  versus  p. 


From  Equation  (7),  if  the  experts  were  perfectly  consistent, 
then  p  =  (^i^)  over  the  entire  range  of  p  and  all  the  data  points  would 


lie  on  the  horizontal  axis.  However,  in  Figure  18  it  is  evident  that  on 

average. 

if  p  >  0.5, 

then  p  >  , 

(12) 

If  p  <  0.5, 

then  P<(v;_v:)' 

(13) 

if  psO.5, 

then  p  =  J  . 

(14) 
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Apparently  there  is  a  behavioral  phenomenon  that  caused 
the  experts  to  respond  inconsistently. 

Consider  the  left  decision  sapling  in  Figure  1 6,  except  this  time 
the  result  node  outcomes  are  measured  by  option  utility  values,  U(i),  U(b), 
and  U(v/),  instead  of  the  options  themselves,  for  the  intermediate,  best, 
and  v/orst  option  utility  values  from  top-to-bottom,  respectively.  Modifying 
Equation  (4)  by  substituting  Vj,  and  with  U(i).  U(b),  and  U(w), 
respectively,  we  get 


U(i)  =  pU(b)  +  (1  -  p)U(w),  thus 

r_u(i>^^  (15) 

P  VU(b)-U(w)>'  •  ' 


The  relationship  between  Equations  (12)  and  (15)  then  is 


U(i)-U(w)  ^ 
U(b)-U(w)y 


(16) 


The  relationship  between  Equations  (13)  and  (15)  then  is 


f  u(i)-u(w) A  Vj-Vw ^ 

Vu(b)-u(w)y  P  Wb-Vw^ 


(17) 


Figure  1 9  plots  both  a  risk  neutral  utility  function  (line  AB)  and 
a  generic  risk  averse  utility  function  (curve  CD).  Let  V^,  V,,  and  V^,  plotted 
on  the  X-coordinate,  be  the  values  of  the  worst,  intermediate,  and  best 
options,  respectively.  Let  U(w),  Ll(i),  and  U(b)  be  the  utility  values  from  the 
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risk  averse  utility  function  corresponding  to  the  worst,  intermediate,  and 
best  options,  respectfully.  Note  that 


for  oil  W.  such  that  <  V,  <  ,  then 

f  U(i)-U(w)  ^  f  Vj-Vw  ^ 

vu(bhU(w)y  f-*  ^  Wb-Vw^  • 


(18) 


Therefore,  o  person  whose  data  values  of  p,  V^,  V,,  and  support 
Equation  (18)  is  displaying  risk  averse  behavior. 


Figure  1 9.  Plot  of  Risk  Neutral  and  Risk  Averse  Utility  Function  Curves. 


Similarly,  if  the  utility  function  curve  is  convex,  the  inequality  in 
Equation  (18)  reverses  and 


U(i)-U(w)  ^ 
U(b)-U(w)y 


=  P< 


(19) 
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implying  that  data  satisfying  Equation  (19)  indicates  the  person  is 
displaying  risk  prone  behavior. 

From  Equation  (14), 

Vi-V„s0.5K-VJ 

2(V,-VJsV,-V„ 

2Vi  =  Vb  +  Vw, 

Vi  =  (^).  (201 

Similarly,  from  Equation  (13),  for  p  <0.5, 

Vi>pVb  +  (l -p)Vw,  and  (21) 

from  Equation  (12),  for  p  >  0.5  , 


Vi<pVb  +  (l-p)Vw.  (22) 

The  risk  neutral,  risk  prone,  and  risk  averse  regions  of  Figure  1 8 
are  those  that  satisfy  Equations  (20),  (21 ),  and  (22),  respectively.  Figure  20 

displays  the  non-neutral  regions  pictorially.  The  experts,  therefore,  are  risk 
averse  when  p  is  large,  risk  neutral  when  p  e  0.5,  and  risk  prone  when  p  is 
small. 

Equations  (20)  through  (22)  also  make  intuitive  sense. 

Figure  21  shows  a  linear  scale  with  and  marked  towards  either  end, 
and  the  risk  prone,  neutral,  and  averse  regions  labeled.  If  V;  is  at  point  1 , 
the  intermediate  option’s  value  is  not  that  much  greater  than  that  of  the 
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Figure  20.  Risk  Prone  and  Risk  Averse  Regions  of  Figure  18. 

worst  option,  and  the  expert  is  willing  to  accept  a  p  that  results  in  a 
negative  expected  value  for  a  gamble  between  the  best  and  worst 
options,  rather  than  accept  the  intermediate  option  for  certain. 
Conversely,  if  V,  is  at  point  2,  the  intermediate  option's  value  is  almost  as 
great  as  the  best  option’s  value,  and  the  expert  must  have  a  p  large 
enough  to  produce  a  positive  expected  value  in  the  gamble. 


Figure  21 .  Linear  Scale  Risk  Prone,  Neutral,  and  Averse  Regions. 
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In  summary,  the  plot  in  Figure  18  displays  a  distinctive  pattern 
that  was  not  anticipated.  Analyzing  the  plot  reveals  that  the  experts  were 
risk  prone  when  Vj  was  closer  to  than  ,  and  risk  averse  when  V,  was 
closer  to  than  . 

c.  Between  Methods  2  and  3 

The  inter-method  consistency  check  between  Methods  2 
and  3  displayed  the  same  gross  parameter  behavior  as  with  the 
preceding  check.  The  value  of  d^^,^  for  this  check,  the  largest  of  the  five 
main  data  subsets,  was  0.3962,  which  appears  to  indicate  a  very  low  level 
of  response  consistency  between  these  two  methods.  The  inter-method 
bias  was  strongly  negative  at  (-0.2052).  Figure  22  replots  the  data  for  this 
inter-method  check  with  p  plotted  on  the  horizontal  axis,  and  p  - 

plotted  on  the  vertical  axis. 


Figure  22.  Alternate  Inter-Methods  2  and  3  Check: 
P-Kriw-llArbw-l)]  versus  p. 


As  with  the  inter-method  consistency  check  between 
Methods  1  and  3,  this  check  also  displayed  the  curious  behavior  of  the 
bias  seemingly  becoming  more  negative  with  increasing  p.  Similar 

arguments  can  be  made  to  show  that,  on  average,  the  experts  were  risk 
prone,  neutrai,  and  averse  when  p  <  0.5,  p  =  0.5,  and  p  >  0.5,  respectively. 

B.  METHOD  APPLICABILITY 

Analysts  using  the  methods  described  in  and  employed  to  conduct 
an  experiment  for  this  thesis  could  find  applicability  in  numerous 
qualitative  situations.  The  methods  are  not  substitutes  for  thorough 
cost-benefit  or  similar  analyses.  However,  in  situations  where  there  is  a 
readily  available  pool  of  subject  matter  experts,  but  not  an  available, 
affordable,  or  computationally  logical  data  base,  these  methods  show 
definite  applicability  for  developing  a  quantifiable  and  comparable 
measure  of  value,  preference,  worth,  etc. 

Given  several  decision  alternative  sets,  whether  they  are  tanks, 
bicycles,  or  televisions,  one  could  relatively  quickly  dissect  the  alternative 
sets  into  attributes  and  attribute  choices.  The  survey  question  formats 
exercised  for  this  thesis  could  then  be  adapted  for  the  particular 
alternative  sets  under  evaluation. 

C.  EXPERIMENT  CONCLUSIONS 

The  procedures  used  in  this  thesis  can  be  employed  to 
quantitatively  evaluate  attribute  choices  with  no  natural  measurement 
basis,  the  first  major  step  in  evaluating  multi-attribute  decision  alternatives. 

The  intra-method  consistency  checks  of  the  experimental  data 
showed  that  the  pairwise  comparison  method  (Method  2)  yielded  notably 
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smaller  dispersion  parameters  than  the  indifference  probability  method 
(Method  3).  Method  2  responses,  however,  were  much  more  biased  than 
those  of  Method  3.  These  general  observations  imply  there  may  be 
possible  tradeoffs  available  between  dispersion  and  bias  between 
Methods  2  and  3.  Consistency  parameters  for  checks  between  methods 
were  generally  greater  than  the  intra-method  parameters. 

The  most  interesting  observation  borne  out  by  variants  of  the 
consistency  checks  was  not  anticipated  before  conducting  the  checks. 
This  was  the  incontestable  relationship  between  expert  response  behavior 
and  attribute  choice  and  indifference  probability  values.  The  experts 
displayed  risk  prone  behavior  when  they  perceived  there  was  little  to 
loose  and  much  to  gain.  Conversely,  the  experts  displayed  risk  averse 
behavior  when  they  perceived  there  was  much  to  loose  and  little  to  gain. 

D.  AREAS  FOR  FUTURE  STUDY 

A  considerable  amount  of  time  spent  on  this  thesis  was  devoted  to 
identifying  the  experiment  subject  matter,  approach,  and  procedures, 
and  survey  development,  trial  testing,  and  modification.  The  time 
devoted  to  experimental  approach  and  survey  development  preempted 
the  thorough  discussion  and  consistency  analysis  of  attribute  weighting 
factors.  The  first  and  foremost  area  of  future  study  would  be  to  develop 
the  discussion  and  analysis  of  attribute  weighting  factors,  analyze  the 
remaining  data  in  Appendix  C,  and  develop  actual  alternative  set  values. 


56 


APPENDIX  A.  AHRIBUTE,  AHRIBUTE  CHOICE,  AND  MULTI-AHRIBUTE 
DECISION  ALTERNATIVE  SET  NOMENCLATURE  AND  INDEXING 

This  appendix  provides  a  detailed  explanation  of  the  nomenclature 
and  indexing  system  used  throughout  this  thesis.  Equation  (1)  [see 
Chapter  II,  Section  B]  is  presented  belov^  as  a  springboard  for  the 
following  discussion: 

V(j)  =  Lbi  Oj  V,  0)  tor  any  j  =  1 , 2, m,  where  (1 ) 

V(j)  =  overall  value  for  alternative  set  j; 

ttj  =  weighting  factor  for  attribute  i;  a,  is  the  amount  of  set  j's 
overall  value  obtained  from  a  unit  of  attribute  i: 

(Xj  >  0  If,  for  attribute  i,  bigger  is  better  [Ref.  1 ,  Table  7.2]:  and, 
ttj  <  0  If,  for  attribute  i,  smaller  is  better:  and, 

V,(j)  =  value  of  attribute  i  for  alternative  set  j. 

DEFINITIONS: 

•Multi-Attribute  Decision  Alternative  Set:  Also  called  a  decision 
alternative,  alternative  set,  or  set,  a  multi-attribute  decision  alternative  set 
is  a  complex  object  upon  which  a  decision  maker  wants  to  place  a 
quantitative  value  in  order  to  directly  compare  it  with  similar  complex 
objects.  For  example,  an  auto  dealership  may  present  a  perspective  car 
buyer  (i.e.,  a  decision  maker)  with  dozens  of  alternative  sets  (i.e.,  cars)  that 
he  wants  to  evaluate. 

•Attribute:  An  attribute  Is  an  alternative  set  characteristic  common 
to  all  (or  at  least  most)  of  the  sets  considered  for  evaluation  and  ranking. 
For  example,  attribute  candidates  for  new  cars  could  include  gas 
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mileage  and  exterior  color.  Note  that  some  attributes  (e.g.,  gas  mileage) 
are  much  easier  to  quantify  than  others  (exterior  color). 

•Attribute  Choice:  An  attribute  choice,  or  choice,  is  the  particular 
item  or  characteristic  of  a  certain  attribute  in  each  alternative  set.  For  the 
car  example,  attribute  choices  for  the  exterior  color  attribute  may  be 
green,  red,  and  black  for  some  car  models,  and  silver,  dark  blue,  red,  and 
turquoise  for  others.  The  gas  mileage  attribute  choice  for  one  model  of 
car  might  be  1 7.9  miles  per  gallon  (mpg)  and  24.1  mpg  for  a  different 
model.  Note  that  the  number  of  attribute  choices  need  not  be  the  same 
for  each  attribute  (six  exterior  color  attribute  choices,  but  only  two  gas 
mileage  choices).  And  that  not  all  alternative  sets  may  be  available  or 
feasible  (the  dealer  may  not  have  a  red  car  that  gets  24.1  mpg). 
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Table  A.l  below  demonstrates  the  architecture  between  attribute 
categories  and  attribute  choices. 


Attribute,  indexed  by  i.  For  example: 

-speed  (1),  range  (2),  esthetics  (3) . maintainability  (n) 

Each  Attribute  need  not  have  the  same  number  of  Attribute  Choices. 

The  Attribute  index,  i,  is  only  for  notational  convenience.  There  is  no 
ranking  or  value  associated  v/ith  i.  When  the  letter  i  is  used  as  a  subscript 
for  the  letters  r  or  V,  the  i  does  imply  an  intermediate  ranking. 


Intra-Attribute  Attribute  Choice 
Ranking,  k 


(Most  preferred  level  of  Attribute  I) 


(Second  Most  preferred  level  of  Attribute  i)  2 


(Third  Most  preferred  level  of  Attribute  i)  •  3 


(Least  preferred  level  of  Attribute  i) 


Tabie  A.l .  Attribute  and  Attribute  Choice  Definitions 


Attribute  Choices  (within  Attribute  il.  indexed  by  k  and  ranked 
in  order  from  the  Attribute  Choice  with  the  most  preferred  level 
of  Attribute  i  (v,  in  the  table)  to  the  Attribute  Choice  with  the 
least  preferred  level  of  Attribute  i  (z,  in  the  table). 

Again,  there  need  not  be  the  same  number  of  Attribute 
Choices  for  each  Attribute.  K,  is  the  number  of  Attribute 
Choices  in  Attribute  i. 
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Table  A.2  below  demonstrates  the  architecture  between  attributes 


and  multi-attribute  decision  alternative  sets. 


Attribute,  indexed  by  i.  For  example: 

-speed  (1),  range  (2),  esthetics  (3), maintainability  (n) 

Each  Attribute  need  not  have  the  same  number  of  Attribute  Choices. 

The  Attribute  index,  i,  is  only  for  notational  convenience.  There  is  no  ranking 
or  value  associated  with  i.  When  the  letter  i  is  used  as  a  subscript  for  the 
letters  r  or  V,  the  i  does  imply  an  intermediate  ranking. 


Multi-Attribute  Decision  Alternative  Sets,  j 

(Note:  The  ordering  of  the  alternative  sets  is  arbitrary.  The  text  below 
describes  a  suggested  possible  method  of  ordering  the  sets.) 

1 

CN 

CO 

••• 

Multi-Attribute  Decision  Alternative  Set ,  j  =  1 .  Set  1  contains  the  most 
preferred  Attribute  Choice  from  each  Attribute  (IF  the  set  is  feasible) 

(In  the  table,  j  =  v,,  v^,  V3, vj 

1 

V, 

V2 

V3 

••• 

Vn 

Multi-Attribute  Decision  Alternative  Set ,  j  =  2.  Set  2  contains  the  most 
preferred  Attribute  Choice  from  each  Attribute  except  Attribute  n  (IF 
the  set  is  feasible).  The  set  j  =  2  Attribute  n  element  is  the  second  most 
preferred  Attribute  Choice  from  Attribute  n. 

2 

V, 

V2 

V3 

••• 

Wn 

Multi-Attribute  Decision  Alternative  Set ,  j  =  3.  Set  3  contains  the  most 
preferred  Attribute  Choice  from  each  Attribute  except  Attribute  n  (IF 
the  set  is  feasible).  The  set  j  =  3  Attribute  n  element  is  the  third  most 
preferred  Attribute  Choice  from  Attribute  n. 

3 

V, 

V2 

V3 

••• 

ii 

Xn 

Subsequent  Multi-Attribute  Decision  Alternative  Sets ,  j  =  4, 5 . m  -1 , 

continue  in  a  fashion  similar  to  the  Arabic  numbering  system  (where 
Attribute  n  equates  to  the  one’s  column.  Attribute  n-1  the  ten's 
column,  etc.) 

••• 

••• 

••• 

••• 

M 

Multi-Attribute  Decision  Alternative  Set ,  j  =  m.  Set  m  contains  the  least 
preferred  Attribute  Choice  from  each  Attribute  (IF  the  set  Is  feasible) 
(In  the  table,  j  =  2,,  22,23 . z^) 

m 

Z2 

Z3 

1 

Table  A.2.  Attribute  and  Multi-Attribute  Decision  Alternative  Sets  Definitions 


Multi-Attribute  Decision  AHernatlve  Sets  contain  a  single  Attribute  Choice  from 
each  Attribute.  The  sets  are  indexed  by  j.  Sets  are  ordered  in  some  logical 
fashion.  For  example:  set  j  =  1  contains  the  Attribute  Choices  with  the  most 
preferred  level  of  their  respective  Attribute  (IF  it  is  a  feasible  set!),  while  the 
set,  j  =  m,  contains  the  Attribute  Choices  with  the  least  preferred  level  of  their 
respective  Attribute  (IF  it  is  a  feasible  set!).  A  quasi  Arabic  numbering  system, 
where  Attribute  n  is  the  one’s  column.  Attribute  n-1  is  the  ten's  column,  etc.,  is 
the  recommended  ordering  system. 
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APPENDIX  B.  LAMPS  DECISION  ALTERNATIVES  SURVEY  GUIDE  AND 

RESPONSE  FORM 


LAMPS  Decision  Alternatives  Survey 

Overview:  This  survey  is  being  conducted  to  collect  thesis  data.  The  data  will  be  used  to 
statistically  analyze  methods  for  assessing  and  ranking  decision  alternatives  that  have  no  natural 
basis  for  comparison.  You,  the  "Decision  Maker",  will  be  presented  with  a  series  of  decision 
problems  involving  LAMPS  (Light  Airborne  Multipurpose  System)  helicopter  systems  and  invited 
to  give  both  subjective  and  quantitative  responses. 

Scenario:  All  the  following  questions  should  be  considered  in  context  with  the  overall  mission 
of  USW  (Undersea  Warfare);  redetection  phase  all  the  way  through  to  battle  damage  assessment 
(BDA). 

Step  1:  Subjective  Rankinas:  You  will  be  presented  with  sets  of  three  helicopter  systems  in 
each  of  four  categories:  Weapons,  Navigation,  Communications,  and  Sensors.  Always  keeping 
in  mind  the  given  scenario  (i.e.y  a  complete  USW  mission),  you  are  to  rank  the  three 
helicopter  systems  in  each  category  from  best  to  worst.  Below  is  a  non-LAMPS  example: 


•Example: 

•Scenario:  You  receive  your  current  salary  and  are  out  shopping  for  a  birthday  gift 
for  your  wife  or  girlfriend. 

•"System  Options": 

•$5  gift 
•$20  gift 
•$100  gift 

•Most  people  would  probably  rank  the  "system  options"  for  the  given  scenario  as 
follows: 


$5  gift 
$20  gift 
$100  gift 


•  _ 3 

•  _ 2_ 

• _ 1 


Do  you  understand  the  task?  Do  you  have  any  questions?  □  Page  1  of  response  form. 

□  •••••••••••• 
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Step  Ifa);  Quantitative  Assessment;  Now  that  you  have  ranked  the  three  helicopter  system 
alternatives  in  each  category,  you  will  be  asked  to  give  your  opinion  on  the  relative  value  of  how 
much  better  the  system  you  ranked  "best"  is  over  the  "intermediate"  and  "worst"  systems,  and 
how  much  better  the  "intermediate"  system  is  over  the  "worst"  system,  always  keeping  in  mind 
that  your  responses  should  be  in  context  with  performing  the  given  scenario  of  a  complete 
USW  mission. 


•Simplified  example: 

•Scenario:  You  receive  your  current  salary  and  are  out  shopping  for  a  birthday  gift 
for  your  wife  or  girlfiiend.  Same  "system  options"  as  before: 

•The  (Best)  $100  bill  is  5  times  better  than  the  (Intermediate)  $20  bill  ■ 

•The  (Best)  $100  ^ift  is  5  times  better  than  the  (Intermediate)  $20  flift  ■ 


•The  (Best)  $100  bill  is  20  times  better  than  the  (worso , 
•The  (Best)  $100  a\ft  is  8>  times  better  than  the  (woist)  _ 


•The  (Intermediate)  $20  bill  is  4  times  better  than  the  (Worst)  $5  bill  . 
•The  (Intermediate)  $20  gift  is  2.5  times  better  than  the  (worso  $5  aii 


Do  you  understand  the  task?  Do  you  have  any  questions?  □  Page  2  of  response  form. 

□  •••••••••••• 
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Step  2(b~):  Quantitative  Assessment  (cont.);  The  following  set  of  questions  asks  your  opinions 
on  the  relative  value  of  how  much  worse  the  "worst"  system  is  compared  to  the  "best"  and 
"intermediate"  systems,  and  how  much  worse  the  "intermediate"  system  is  compared  to  the  "best" 
system.  Again,  always  keep  in  mind  that  your  responses  should  be  in  context  with 
performing  the  given  mission  scenario. 


•Simplified  example; 

•Scenario:  You  receive  your  current  salary  and  are  out  shopping  for  a  birthday  gift 
for  your  wife  or  girlfhend.  Same  "hardware  options"  as  before: 


•Thefwotsi)  $5  bill  is  20  times  than  the  (b«i)  $100  bill  . 
•The  (Worst)  $5  gift  is  3  times  worse  than  the  (Best)  $100  . 


•The  (Worst)  $5  bill  is  4  times  worse  than  the  (Intermediate)  $20  bill  . 
•The  (Worst)  $5  is  2.5  times  worse  than  the  (Intermediate)  $20  aiyft 


(Intermediate) 

(Intermediate) 


$20  bill  is  5 
$20  giift  is  5 


times  worse  than  the  (Beso  _ 
_  times  worse  than  the  (Best) 


$100  bill  . 
$100  a\ft 


Do  you  understand  the  task?  Do  you  have  any  questions?  □  Page  3  of  response  form. 

□  •••••••••••• 


Step  3;  Marking  a  Linear  Scale:  You  Avill  now  be  presented  with  a  line  labeled  at  one  end 
with  a  zero  (0)  and  the  other  end  with  a  100.  If  zero  represents  an  absolutely  worthless 
helicopter  system  for  the  given  scenario,  and  100  represents  the  absolute  best  helicopter  system 
imaginable  for  the  given  scenario,  mark  the  line  where  you  think  the  "best"  (B),  "intermediate" 
(I),  and  "worst"  (W)  systems,  as  ranked  earlier,  would  fall  on  the  line. 


►Simplified  example: 

•Scenario:  You  receive  your  current  salary  and  are  out  shopping  for  a  birthday  gift 
for  your  wife  or  girlfnend.  Same  "system  options"  as  before; 


$5  ^ift  $20  0ift 

W  I 

No|gift..| - +1 - +~- - +- 

19  20  1  30 

_ 


$100  gift 
5 

40  i  50  60  70 


■+- 

80 


r  A 

Best  possible  gift 
imaginable,  given 

financial  condition. 

•-k- — — 

90  100 


Do  you  understand  the  task?  Do  you  have  any  questions?  □  Page  4  of  response  form. 

□  •••••••••••• 
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Stgn4ra^:  Let’s  Make  a  Deal!;  You  are  outfitted  with  the  system  you  have  ranked  as 
"intermediate".  It  is  yours  to  employ  in  the  given  scenario.  However,  you  have  the  option  to 
trade  in  your  current  (intermediate)  system  for  a  chance  at  getting  the  "best"  system.  If  you  take 
the  gamble  and  win,  you  get  the  "best"  system.  If  you  take  the  gamble  and  lose,  you  get  the 
"worst"  system.  If  you  do  not  take  the  gamble,  you  keep  the  "intermediate"  system.  What  is  the 
nunimum  percentage  chance  of  winning  the  "best"  system  you  are  willing  to  accept  in  order 
to  take  the  gamble? _ _ _ _ 


•Simplified  example: 

•Scenario;  You  are  a  college  football  coach.  There  are  three  seconds  left  in  the  last  game  of  the 
regular  season.  Your  team  just  scored  a  touchdown  and  is  currently  down  by  one  point.  Your 
"system  options"  are  the  following: 

•"Worst":  Do  not  convert  on  the  extra  point  attempt,  either  kick  or  two-point  attempt,  and  loose 
the  game. 


•"Intermediate" ;  Kick  the  extra  point  to  tie  the  game. 
•"Best";  Score  a  two-point  conversion  to  win  the  game 


•Now,  if  you  are  guaranteed  able  to  tie  the  game  (your  team  has  an  outstanding  kicker,  he  never 
misses),  what  is  the  minimum  percentage  chance  of  getting  the  "best"  result  (i.e.,  win^g  the 
game)  you  are  willing  to  accept  in  order  to  take  the  gamble  and  possibly  end  up  with  the  worst  result 
(i.e.,  loosing  the  game)? 

•Your  minimum  acceptable  percentage  chance  of  getting  the  "best"  result  would  most  likely  depend 
on  such  factors  as  your  win-lose  record,  post-season  bowl  game  opportunities,  or  chances  of  winning 
the  league  or  national  title,  etc.  For  example; 


•If  tying  the  game  meant  holding  on  to  the  league  championship  and  a  bowl  game  appearance, 
while  loosing  the  game  would  terminate  your  season,  the  minimum  percentage  chance  of  getting  the 
"best"  result  (i.e.,  winning  the  game)  would  probably  have  to  be  very  high  (close  to  100  percent)  in 
order  to  take  the  gamble. 

•On  the  other  hand,  if  the  only  way  your  team  could  get  a  bowl  game  appearance  was  to  wm 
the  game  outright,  you  would  probably  be  willing  to  take  a  big  gamble,  risk  the  guaranteed  tie,  and 
accept  a  rather  low  minimum  percentage  chance  (close  to  zero  percent)  of  getting  the  "best"  result 
(winning  the  game). 


•The  previous  examples  are  probably  the  extremes.  Depending  on  the  consequences  of  a  win, 
tie  or  lose  (e.g.,  school  pride,  coaches  job,  etc.),  your  minimum  acceptable  percentage  chance  of 
risking  the  guaranteed  "intermediate"  result  for  a  chance  at  getting  the  "best"  result  would  likely  fall 
somewhere  in  between. 


For  the  actual  survey  questions,  you  will  be  asked  to  provide  a  percent  figure  (from  0-100, 
inclusive) .  Remember  to  keep  in  mind  the  scenario,  a  complete  USW  mission. 

Do  you  understand  the  task?  Do  you  have  any  questions?  □  Page  5  of  response  form. 

m  .  .  •  •  •  •  •  •  •  •  •  • 
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Step  4rb^;  Let’s  Make  a  Deal!  (Again!):  Just  as  with  the  previous  task,  you  are  outfitted  with 
the  system  you  have  ranked  as  "intermediate".  It  is  yours  to  employ  in  the  given  scenario. 
However,  you  have  the  option  to  trade  in  your  current  (intermediate)  system  for  a  chance  at 
getting  the  "best"  system.  If  you  take  the  gamble  and  win,  you  get  the  "best"  system.  If  you  take 
the  gamble  and  lose,  you  get  the  "worst"  system.  If  you  do  not  take  the  gamble,  you  keep  the 
"intermediate"  system.  What  is  the  maximum  percentage  chance  of  getting  the  "worst"  system 
you  are  willing  to  accept  in  order  to  take  the  gamble? 


•Simplified  example; 

•Scenario;  Same  sample  scenario  as  before...  You  are  a  college  football  coach.  There  are  three 
seconds  left  in  the  last  regular  game  of  the  season.  Your  team  just  scored  a  touchdown  and  is 
currently  down  by  one  point.  Your  "system  options"  are  the  following; 


•"Worst";  Do  not  convert  on  the  extra  point  attempt,  either  kick  or  two-point  attempt,  and  loose 
the  game. 

•  "Intermediate" ;  Kick  the  extra  point  to  tie  the  game. 

•"Best";  Score  a  two-point  conversion  to  win  the  game 


•Now,  if  you  are  guaranteed  able  to  tie  the  game  (your  team  has  an  outstanding  kicker,  he  never 
misses),  what  is  the  maximum  percentage  chance  of  getting  the  "worst"  result  (i.e.,  loosing  the 
game)  you  are  willing  to  accept  in  order  risk  the  guaranteed  tie  gambling  for  the  win? 

•As  with  the  previous  task,  your  maximum  acceptable  percentage  chance  of  getting  the  "worst" 
result  would  most  likely  depend  on  such  &ctors  as  your  win-lose  record,  post-season  bowl  game 
opportunities,  or  chances  of  winning  the  league  or  national  title,  etc.  For  example; 

•If  tying  the  game  meant  holding  on  to  the  league  championship  and  a  bowl  game  appearance, 
while  loosing  the  game  would  terminate  your  season,  the  maximum  percentage  chance  of  getting  the 
"worst"  result  (i.e.,  loosing  the  game)  would  probably  have  to  be  very  small  (close  to  zero  percent)  in 
order  to  take  the  gamble. 

•On  the  other  hand,  if  the  only  way  your  team  could  get  a  bowl  game  appearance  was  to  win 
the  game  outright,  you  would  probably  be  willing  to  take  a  big  gamble,  risk  the  guaranteed  tie,  and 
accept  a  rather  high  maximum  percentage  chance  (close  to  100  percent)  of  getting  the  "worst"  result 
(i.e.,  loosing  the  game). 


•The  previous  examples  are  probably  the  extremes.  Depending  on  the  consequences  of  a  win, 
tie,  or  lose  (e.g.,  school  pride,  coaches  job  security,  etc.),  your  maximum  acceptable  percentage 
chance  of  risking  the  guaranteed  "intermediate"  result  gambling  for  the  "best"  result  would  likely  fell 
somewhere  in  between. 


For  the  actual  survey  questions,  you  will  be  asked  to  provide  a  percent  figure  (fi'om  0-100, 
inclusive) .  Remember  to  keep  in  mind  the  scenario,  a  complete  USW  mission. 

Do  you  understand  the  task?  Do  you  have  any  questions?  □  Page  6  of  response  form. 

□  •••••••••••• 
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Note;  This  final  part  of  the  survey  should  only  take  a  few  moments.  You  will  be  presented  with 
a  short  series  of  questions  soliciting  your  inputs  on  the  relative  importance  of  the  four  categories 
of  systems  presented  earlier;  namely  Weapons,  Communications,  Navigation,  and  Sensors.  The 

underiying  scenario  remains  unchanged;  a  complete  USW  mission  from  the  redetection 
phase  all  the  way  through  BDA. 

Of  course,  all  four  systems  categories  are  vital  to  complete  a  successful  USW  mission. 
These  last  few  questions  hope  to  canvass  your  expert  opinion  as  to  the  relative  importance  of 
each. 

□  •  •  •  •  •  •  •  •  •  •  •  • 

Step  5;  Subjective  Rankines;  The  four  helicopter  systems  categories,  Weapons,  Navigation, 
Communications,  and  Sensors,  are  to  be  ranked  from  most  important  (or  valuable)  to  least 
important  (or  valuable)  with  respect  to  the  given  mission  scenario  (i.e.,  a  complete  USW 
mission). 

- 

Weapons  • _ 

Navigation  • _ 

Communications  • _ 

Sensors  • _ 

V _ y 

Do  you  understand  the  task?  Do  you  have  any  questions?  □  Page  7  of  response  form. 

□  •••••••••••• 


Step  6;  Pflirwisg  Comparisons;  You  will  now  be  asked  to  give  your  opinions  on  the  relative 
value  (or  importance)  to  the  given  mission  scenario  (i.e.,  a  complete  USW  mission)  of  the 
four  systems  categories,  compared  two  at  a  time.  Your  answers  will  be  placed  in  a  partial  matrix. 
Entries  in  the  matrix  will  be  numerical  values  of  how  much  more  important  (or  valuable)  the  row 
system  category  is  relative  to  the  column  system  category.  For  example,  if  the  row  system 
category  is  10  times  more  valuable  (important)  for  the  USW  mission  than  the  column  system 
category,  you  would  enter  a  10.  If,  on  the  other  hand,  the  column  category  was  three  times  more 
important  than  the  row  category,  you  would  enter  1/3. 


(1)[ 

L 

]■ 

Do  you  understand  the  task? 

(2)[ 

3 

Do  you  have  any  questions? 

(3)[ 

] 

□  Page  7  of  response  form. 

k _ 

(2)  (3)  (4) 


][ 


][ 


□ 
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Step  7:  Unlimited  Budget:  You  are  now  outfitted  with  the  systems  you  ranked  as  "worst" 
during  the  first  part  of  the  survey  for  all  four  of  the  system  categories.  If  you  were  provided  a 
budget  that  allowed  you  improve  the  systems  from  "worst"  to  "best"  in  all  four  categories,  what 
proportion  (percentage)  of  the  budget  would  you  allocate  to  each  categoryf!  (Keep  in  mind  the 
£iven  scenario,  i.e.,  a  complete  USW  mission.) 

(  ^ 

Weapons 

Navigation 

Communications 

Sensors - - ' 

TOTAL _ 100  j 

Do  you  understand  the  task?  Do  you  have  any  questions?  □  Page  7  of  response  form. 


Step  8:  Let’s  Really  Make  a  Deal!:  You  are  outfitted  with  the  "best"  system  in  one  system 
category,  and  the  "worst"  systems  in  the  remaining  three  system  categories.  This  system  suite  is 
yours  to  employ  in  the  given  scenario  (complete  USW  mission).  However,  you  have  the 
option  to  trade  in  your  current  (one  "best"  and  three  "worsts")  suite  for  a  chance  at  getting  the 
"best"  systems  in  all  four  system  categories.  If  you  take  the  gamble  and  win,  you  get  the  "best 
systems  in  all  four  system  categories.  The  downside  is,  if  you  take  the  gamble  and  lose,  you  get 
the  "worst"  systems  in  all  four  categories.  If  you  do  not  take  the  gamble,  you  keep  the  system 
suite  you  have  been  given  (one  "best"  and  three  "worsts").  What  is  the  ntinimunt  percentage 
chance  of  winning  the  "best”  systems  in  all  four  system  categories  you  are  willing  to  accept  in 
order  to  take  the  gamble? 


Weapon 


Comms  Sensors 


Given  System  Suite. 
Win  Gamble . 


Lose  Gamble. 


^Minimum  percentage  chance  of  winmng  in  order  to  take  gamble: 


Do  you  understand  the  task?  Do  you  have  any  questions?  □  Page  8  of  response  form. 


□ 


Step  9:  Decision  Maker  Background  Information;  Please  answer  the  background  information 
questions  listed  on  page  9  of  the  response  form.  This  data  will  be  analyzed  for  any  discernible 
patterns  or  trends. 
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Response  Form 

.  . . 

Step  1;  Subjective  Rankings;  Rank  the  following  sets  of  helicopter  systems  from  best  ( 1 )  to 
worst  (3).  (Again,  keep  in  mind  the  given  scenario,  i.e.,  a  complete  USW  mission.) 


r 

•Weapons:  1 

•Mk-46  Torpedo  (Mk-46), 

• 

•Conventional  500#  Depth  Charge  (500#), 

• 

•Mk-50  Torpedo  (Mk-50). 

• 

< _ _ 

•Navigation: 

•Global  Positioning  System  (GPS), 

• 

•TACAN  (TACAN), 

• 

•Doppler  Radar  (Dop). 

• 

— 

•Communications: 

•Datalink  (D/L), 

• 

•Satellite  Communications  (Sat  Com), 

• 

•UHFATIF  Radio  (Radio). 

• 

. . J 

/ - ^ 

•Sensors: 

•Magnetic  Anomaly  Detector  (MAD), 
•Surface  Search  Radar  (Radar), 
•Forward  Looking  Infrared  (FLIR). 


Step  2fa^:  Quantitative  Assessment;  Give  your  quantitative  opinions  on  how  much  better  the 
"best”  system  is  over  the  "intermediate"  and  "worst"  systems,  and  how  much  better  the 
"intermediate"  system  is  over  the  "worst".  (Again,  keep  in  mind  the  given  scenario,  i.e.,  a 
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Step  2(b>;  Quantitative  Assessment  fcont.);  Give  your  quantitative  opinions  on  how  much 
worse  the  "worst"  system  is  than  the  "best"  and  "intermediate"  systems,  and  how  much  worse  the 
"intermediate"  system  is  than  the  "best".  (Again,  keep  in  mind  the  given  scenario,  i.e.,  a 
complete  USW  mission.) 

□  •••••••••••• 


r .  '  '  . . 

•Weapons:  Best; 

\ 

Intermediate;  Worst; 

•The  (Worst) 

is 

times  worse  than  the  (Best) 

•The  rwoist) 

is 

times  worse  than  the  (intennediate) 

•The  (Intermediate) 

is 

times  worse  than  the  (Beso 

r 

\ 

•Navigation:  Best; 

Intermediate;  Worst; 

•The  (Worst) 

is 

times  worse  than  the  (Best) 

•The  fWorst) 

is 

times  worse  than  the  (intennediate) 

•The  (Intennediate) 

is 

times  worse  than  the  (Best) 

_ _ _ _ y 

A 

•Communications: 

Best; 

Intermediate;  Worst; 

•The  (Worst) 

is 

times  worse  than  the  (Beso 

•The  (Worst) 

is 

times  worse  than  the  (intennediate) 

•The  (Intermediate) 

is 

times  worse  than  the  (Best) 

J 

A 

•Sensors:  Best: 

Intermediate;  Worst; 

•The  (Worst) 

is 

times  worse  than  the  (Best) 

•The 


(Worst) 


IS 


times  worse  than  the  (Intennediate)  _ 


•The  (Intermediate) 


IS 


times  worse  than  the  (Best) 


Step  3:  Marking  a  Linear  Scale;  If  zero  represents  an  absolutely  worthless  system  for  the 
given  scenario,  and  100  represents  the  absolute  best  system  imaginable  for  the  given  scenario, 
mark  the  line  where  you  think  the  "best"  (B),  "intermediate"  (I),  and  "worst"  (W)  systems  for 
each  category,  as  ranked  earlier,  would  fall  on  the  line.  (Keep  in  mind  the  given  scenario,  i.e., 
a  complete  USW  mission.) 

□  •••••••••••• 


•Weapons:  Worst:. 


Intermediate: 


Worthless 

system 


10  20 


70  80 


Best  system  I 
imaginable  ; 


•Navigation:  Worst: 


Intermediate: 


Worthless 

system 


ll  1 

-j _ 

_ 

1  Best  system 

Jl  -  + 

^0  10 

20 

30 

40 

50 

60 

70 

80 

90 

1  imaginable 

lod— — ^ 

•Communications:  Worst:. 


Intermediate: 


Worthless 

system 


10  20 


40  50 


70  80 


Best  system 
imaginable 
100— — :: 


•Sensors:  Worst:. 


Intermediate: 


Worthless 

system 


10 


40  50 


70  80 


Best  system 
imaginable 
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Steo  4(a^;  Let's  Make  a  Deal!:  You  are  outfitted  with  the  system  you  have  ranked  as 


"intermediate".  It  is  yours  to  employ  in  the  given  scenario.  However,  you  have  the  option  to 
trade  in  your  current  (intermediate)  system  for  a  chance  at  getting  the  "best"  system.  If  you  take 
the  gamble  and  win,  you  get  the  "best"  system.  If  you  take  the  gamble  and  lose,  you  get  the 
"worst"  system.  If  you  do  not  take  the  gamble,  you  keep  the  "intermediate"  system.  What  is  the 
minimum  percentage  chance  of  winning  the  "best"  system  you  are  willing  to  accept  in  order 
to  take  the  gamble?  (Keep  in  mind  the  given  scenario,  i.e.,  a  complete  USW  mission.) 

□  •  •  •  •  •  •  •  •  •  •  •  • 
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Step  5;  Subjective  Rankings;  The  four  helicopter  systems  categories.  Weapons,  Navigation, 
Communications,  and  Sensors,  are  to  be  ranked  (1)  to  (4)  from  most  important  (or  valuable)  to 
least  important  (or  valuable)  with  respect  to  the  given  mission  scenario  (i.e.,  a  complete  USW 
mission). 


□  •  •  •  •  •  •  •  •  •  •  •  • 

Step  6;  Pairwise  Comparisons;  You  will  now  be  asked  to  give  your  opinions  on  the  relative 
value  (or  importance)  to  the  given  mission  scenario  (i.e.,  a  complete  USW  mission)  of  the 
four  systems  categories,  compared  two  at  a  time.  Your  answers  will  be  placed  in  a  partial 
matrix.  Entries  in  the  matrix  will  be  numerical  values  of  how  much  more  important  (or  valuable) 
the  raw  system  category  is  relative  to  the  column  system  category.  For  example,  if  the  row 
system  category  is  10  times  more  valuable  (important)  for  the  USW  mission  than  the  column 
system  category,  you  would  enter  a  10.  If,  on  the  other  hand,  the  column  category  was  three 
times  more  important  than  the  row  category,  you  would  enter  1/3. 


□  ••••••••  •••• 

Step  7;  Unlimited  Bud£et:  You  are  now  outfitted  with  the  systems  you  ranked  as  "worst" 
during  the  first  part  of  the  survey  for  all  four  of  the  system  categories.  If  you  were  provided  a 
budget  that  allowed  you  improve  the  systems  from  "worst"  to  "best"  in  all  four  categories,  what 
proportion  (percentage)  of 
the  budget  would  you  allocate  to  each 
category?  (Keep  in  mind  the  given 
scenario,  i.e.,  a  complete  USW  mission.) 


□  ••••••• 
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Step  8;  Let*s  Really  Make  a  Deal!:  You  are  outfitted  with  the  "best"  system  in  one  system 
category,  and  the  "worst"  systems  in  the  remaining  three  system  categories.  This  system  suite  is 
yours  to  employ  in  the  given  scenario  (complete  USW  mission).  However,  you  have  the 
option  to  trade  in  your  current  (one  "best"  and  three  "worsts")  suite  for  a  chance  at  getting  the 
"best"  systems  in  all  four  system  categories.  If  you  take  the  gamble  and  win,  you  get  the  "best" 
systems  in  all  four  system  categories.  The  downside  is,  if  you  take  the  gamble  and  lose,  you  get 
the  "worst"  systems  in  all  four  categories.  If  you  do  not  take  the  gamble,  you  keep  the  system 
suite  you  have  been  given  (one  "best"  and  three  "worsts").  What  is  the  minimum  percentage 
chance  of  winning  the  "best"  systems  in  all  four  system  categories  you  are  willing  to  accept  in 
order  to  take  the  gamble? 


Given  System  Suite.. 

Win  Gamble . 

Lose  Gamble . 


Weapon 

Nav 

Comms 

Sensors 

(Best) 

(Worst) 

(Worst) 

(Worst) 

(Best) 

(Best) 

(Best) 

(Best) 

(Worst) 

(Worst) 

(Worst) 

(Worst) 

♦Minimum  percentage 
chance  of  winning  in 
order  to  take  gamble: 


Given  System  Suite.. 

Win  Gamble . 

Lose  Gamble . 


Weapon 

Nav 

Comms 

Sensors 

(Worst) 

(Best)  ■ 

(Worst) 

(Worst) 

(Best) 

(Best) 

(Best) 

(Best) 

(Worst) . 

(Worst) 

(Worst) . . 

(Worst)  ■ 

♦Minimum  percentage 
chance  of  winning  in 
order  to  take  gamble: 


Given  System  Suite. 

Win  Gamble . 

Lose  Gamble . 


Weapon 

Nav 

Comms 

Sensors 

(Worst) 

(Worst) 

(Best) 

(Worst) 

(Best) 

(Best) 

(Best) 

(Best) 

(Worst) 

(Worst) 

(Worst) 

(Worst) 

♦Minimum  percentage 
chance  of  winning  in 
order  to  take  gamble: 


Given  System  Suite. 

Win  Gamble . 

Lose  Gamble . 


Weapon 

Nav 

Comms 

Sensors 

(Worst) 

(Worst) 

(Worst) 

(Best) 

(Best) 

(Best) 

(Best) 

(Best) 

(Worst) 

(Worst) 

(Worst) 

(Worst) 

♦Minimum  percentage 
chance  of  winning  in 
order  to  take  gamble: 


75 


Step  9;  Decision  Maker  Background  Information; 


•Community  /  years-months  experience  (include  FRS);  •Mk-I  / _  •Mk-III  / 

•Other  military  aviation  experience: _ 

•Year  Group: _ 


•LAMPS  flight  hours. 


•Total  flight  hours  (military): 


•Months  LAMPS  at  sea  time: _ 

•LAMPS  FRS:  •Mayport 

•North  Island 
•Other : 


•Have  you  ever  worked  and/or  flown  with: 


•Depth  Charges:  Y/N 

•GPS:  Y/N 

•Sat  Com:  Y/N 

•FLIR:  Y/N 


•LAMPS  Homeport(s):  •Mayport  •SWATS  Grad?: 

(Excluding  FRS)  •North  Island 

•Barbers  Point 
•Atsugi 

•Other(s): _ 


•Type  platform(s)  on  which  conducted  LAMPS  cruise: 
•FFG-7  •DD-1052 

•DD-963  •Other(s): _ 

•CG-47  • 


Y/N 


•"Actual"  USW  time  (hours)  on  "non  friendly"  submarine(s): _ 

•Any  special  tactics  schools  you  have  attended  (e.g.,  SWATS): _ 

•Experience  with  experimental  equipment  /  R&D  tests  for  LAMPS  applications  (e.g.,  LIDAR, 
Magic  Lantern,  FLIR,  RDP  (Radar  Data  Processor),  Penguin,  etc.)  (Keep  it  UNCLAS  please): 


•Any  comments  on  survey  as  a  whole  (No  holds  barred!) 


•What  other  background  questions  should  I  have  asked?: 
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APPENDIX  C.  SUMMARY  TABLE  OF  RAW  DATA 
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Survey  step  6:  PAIRWISE  COMPARISONS  (Attribute) 
I  (Row  #),  (Column  #)  I  I 
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Survey  Step  7;  UNLIMITED  BUDGET _ Survey  Step  8:  LETS  REALLY  MAKE  A  PEA 

Weaps  I  Nav  Comms  |  Sensor  |  Weaps  |  Nav  Comms  |  Sensor  _ 
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One  Minus  "Second"  Indifference  Probabilit 
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One  Minus  "Second"  Indifference  Probability,  "Second"  Indifference  Probability,  1-q 

Communications 
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n  "Best  vs  Intermediate"  Pairwise  Comparison  Ratio,  ru 

^  Weapons,  Navigation,  Communications,  Sensors 


Figure  A.D-9 
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Navigation  Attribute  Marking  A  Linear  Scale  Ratios,  attribute  Marking  A  Linear  Scale  Ratios. 


Weapon  Attribute  Pairwise  Comparison  Ratios, 

rUt  fbm  Tiw 


Figure  A.D-12 
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Navigation  Attribute  Pairwise  Comparison  Ratios, 

riM,  rbw,  riw 


Figure  A.D-13 


91 


92 


Attribute  Marking  A  Linear  Scale  Calculated 
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Communications  Attribute  Marking  A  Linear  Scale 
Calcuiated  indifference  Probability,  (Vi  -  VvJ/(Vb  -  Vw) 
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